1 00:00:00,000 --> 00:00:01,930 PROFESSOR: --wind up doing a substantial part of Chapter 2 00:00:01,930 --> 00:00:04,000 11, although I do tend to be over-optimistic. 3 00:00:04,000 --> 00:00:06,810 4 00:00:06,810 --> 00:00:11,830 So we're handing out Chapter 11, which I expect to revise 5 00:00:11,830 --> 00:00:13,810 still further. 6 00:00:13,810 --> 00:00:17,110 As we get towards these end chapters, these are things 7 00:00:17,110 --> 00:00:21,820 I've only taught from, at most, once before, and so I 8 00:00:21,820 --> 00:00:24,210 have more revisions to do than the earlier chapters which 9 00:00:24,210 --> 00:00:27,580 I've used for many years. 10 00:00:27,580 --> 00:00:31,660 The Problem Set Seven, as we announced last time, you only 11 00:00:31,660 --> 00:00:34,010 needed to hand in the first four problems today, so we're 12 00:00:34,010 --> 00:00:37,330 only handing out the solutions for the first four today. 13 00:00:37,330 --> 00:00:40,090 I haven't made up any additional problems, so 14 00:00:40,090 --> 00:00:43,350 Problem Set Eight is simply to do the last two problems in 15 00:00:43,350 --> 00:00:46,490 Problem Set Seven, due next Wednesday. 16 00:00:46,490 --> 00:00:50,130 And a reminder that we have no class on Monday because of 17 00:00:50,130 --> 00:00:53,870 this peculiar local holiday we have called Patriot's Day, 18 00:00:53,870 --> 00:00:54,830 when we run the marathon. 19 00:00:54,830 --> 00:00:57,070 And I hope you'll all be out there running the marathon. 20 00:00:57,070 --> 00:01:00,400 21 00:01:00,400 --> 00:01:02,095 So enjoy the long weekend. 22 00:01:02,095 --> 00:01:05,330 23 00:01:05,330 --> 00:01:08,330 So what have we been doing in Chapter 10? 24 00:01:08,330 --> 00:01:18,230 We've basically been getting minimal trellis realizations, 25 00:01:18,230 --> 00:01:22,880 or state-state space realizations, in system theory 26 00:01:22,880 --> 00:01:29,960 terms, for linear block codes. 27 00:01:29,960 --> 00:01:36,670 We've been focusing on binary linear block codes, but 28 00:01:36,670 --> 00:01:43,950 clearly everything we've been doing goes to block codes over 29 00:01:43,950 --> 00:01:46,160 any finite field or, in fact, over the 30 00:01:46,160 --> 00:01:47,795 real or complex field. 31 00:01:47,795 --> 00:01:49,960 It's just basic realization theory, 32 00:01:49,960 --> 00:01:53,790 minimal realization theory. 33 00:01:53,790 --> 00:02:00,060 So as long as we have a linear time-varying system, it works. 34 00:02:00,060 --> 00:02:02,090 I had a question after class last time, well, doesn't it 35 00:02:02,090 --> 00:02:04,050 work just as well for nonlinear systems? 36 00:02:04,050 --> 00:02:07,720 Can't we do the trellis in the same way? 37 00:02:07,720 --> 00:02:08,990 In general, not. 38 00:02:08,990 --> 00:02:10,919 If it's a nonlinear system with a group 39 00:02:10,919 --> 00:02:12,585 property, then you can't. 40 00:02:12,585 --> 00:02:14,660 It's really the group property that's key to these 41 00:02:14,660 --> 00:02:17,260 constructions. 42 00:02:17,260 --> 00:02:20,320 And we seem to have pretty well cracked the problem. 43 00:02:20,320 --> 00:02:23,190 We have this tool called the trellis-oriented or 44 00:02:23,190 --> 00:02:26,830 minimal-span generator matrix, from which we can read all the 45 00:02:26,830 --> 00:02:28,540 parameters of the trellis, from which we 46 00:02:28,540 --> 00:02:30,430 can construct a trellis. 47 00:02:30,430 --> 00:02:34,240 Really, we found that the trellis-oriented generator 48 00:02:34,240 --> 00:02:37,950 matrix contains all the information we need to 49 00:02:37,950 --> 00:02:41,360 construct a minimal trellis. 50 00:02:41,360 --> 00:02:42,960 And it's just a matter of turning the crank 51 00:02:42,960 --> 00:02:44,210 once we have that. 52 00:02:44,210 --> 00:02:46,620 53 00:02:46,620 --> 00:02:52,230 The only degree of freedom that I've allowed so far in 54 00:02:52,230 --> 00:02:56,240 trellis constructions is sectionalization. 55 00:02:56,240 --> 00:03:01,200 I've shown you that if we can choose where to put the state 56 00:03:01,200 --> 00:03:02,450 spaces wherever we like-- 57 00:03:02,450 --> 00:03:05,210 58 00:03:05,210 --> 00:03:07,990 typically we put them in the center and other places, 59 00:03:07,990 --> 00:03:10,040 sometimes we put them everywhere we could put them, 60 00:03:10,040 --> 00:03:13,400 that's called an unsectionalized trellis-- 61 00:03:13,400 --> 00:03:19,570 and by so doing, we can sometimes reduce the apparent 62 00:03:19,570 --> 00:03:21,520 state-space complexity. 63 00:03:21,520 --> 00:03:25,340 But we can never reduce the branch complexity. 64 00:03:25,340 --> 00:03:30,030 So we're left with branch complexity as a fundamental 65 00:03:30,030 --> 00:03:32,520 complexity parameter for linear block 66 00:03:32,520 --> 00:03:35,890 codes, or so it seems. 67 00:03:35,890 --> 00:03:37,320 Is there any other-- 68 00:03:37,320 --> 00:03:38,141 yes? 69 00:03:38,141 --> 00:03:42,470 AUDIENCE: Does the complexity of the number of computations 70 00:03:42,470 --> 00:03:45,452 you need to go through the trellis depends only on the 71 00:03:45,452 --> 00:03:46,856 number of branches, right? 72 00:03:46,856 --> 00:03:48,640 PROFESSOR: It depends primarily 73 00:03:48,640 --> 00:03:50,580 on the number branches. 74 00:03:50,580 --> 00:03:52,050 It also depends on the edges. 75 00:03:52,050 --> 00:03:59,140 If you go through a detailed operation count, I think I 76 00:03:59,140 --> 00:04:03,540 give the formula for if you count additions and 77 00:04:03,540 --> 00:04:08,130 comparisons as an addition and selection you don't count as a 78 00:04:08,130 --> 00:04:12,580 logical operation, then I think the total count for the 79 00:04:12,580 --> 00:04:17,630 Viterbi algorithm is twice the number of edges in the graph, 80 00:04:17,630 --> 00:04:20,529 which is the number of branches, minus the number of 81 00:04:20,529 --> 00:04:24,944 vertices, which you lose one for every-- 82 00:04:24,944 --> 00:04:26,375 AUDIENCE: What do you mean by vertices? 83 00:04:26,375 --> 00:04:28,960 PROFESSOR: By vertices I mean states, now I'm talking about 84 00:04:28,960 --> 00:04:30,430 it as a graph. 85 00:04:30,430 --> 00:04:32,630 Minus one or plus one or something. 86 00:04:32,630 --> 00:04:35,220 So that's an exact count for the Viterbi algorithm. 87 00:04:35,220 --> 00:04:39,560 But it's dominated by the total number of edges in the 88 00:04:39,560 --> 00:04:42,990 graph which, in turn, is well-estimated by the maximum 89 00:04:42,990 --> 00:04:45,020 branch complexity at any one time. 90 00:04:45,020 --> 00:04:47,570 91 00:04:47,570 --> 00:04:51,640 So there's a bit of a trade-off between an exact 92 00:04:51,640 --> 00:04:57,676 count and just getting a gross handle on the situation. 93 00:04:57,676 --> 00:05:01,760 But I say it's certainly got to exceed the branch 94 00:05:01,760 --> 00:05:02,400 complexity. 95 00:05:02,400 --> 00:05:05,170 So when I find the branch complexity goes up 96 00:05:05,170 --> 00:05:08,650 exponentially, even for one branch, then I know that the 97 00:05:08,650 --> 00:05:10,950 total Viterbi algorithm complexity has to go up 98 00:05:10,950 --> 00:05:14,462 exponentially with that. 99 00:05:14,462 --> 00:05:16,805 All right? 100 00:05:16,805 --> 00:05:20,160 So is that the whole story? 101 00:05:20,160 --> 00:05:23,650 Do we now have a pretty definitive handle on the, 102 00:05:23,650 --> 00:05:28,200 quote, complexity of at least maximum likelihood decoding, 103 00:05:28,200 --> 00:05:31,800 trellis decoding of a block code. 104 00:05:31,800 --> 00:05:35,790 And if you've read Chapter 11, you know there's one other 105 00:05:35,790 --> 00:05:38,940 degree of freedom that we haven't exploited yet in 106 00:05:38,940 --> 00:05:41,730 representing a linear block code. 107 00:05:41,730 --> 00:05:44,570 Anyone read that far, or are you just going to wait for me 108 00:05:44,570 --> 00:05:46,720 to explain it to you? 109 00:05:46,720 --> 00:05:49,010 Flip the coordinates, very good. 110 00:05:49,010 --> 00:05:51,440 Could that make a difference? 111 00:05:51,440 --> 00:05:53,050 Yes, it certainly could. 112 00:05:53,050 --> 00:06:04,590 Let's take our standard example by now, the 8-4-4 code 113 00:06:04,590 --> 00:06:07,090 with trellis-oriented matrix that looks like this. 114 00:06:07,090 --> 00:06:10,030 115 00:06:10,030 --> 00:06:14,370 OK, and we know everything about this by now. 116 00:06:14,370 --> 00:06:18,510 But suppose, say, we flip these two center coordinates. 117 00:06:18,510 --> 00:06:25,070 Certainly it would be an equivalent code from all of 118 00:06:25,070 --> 00:06:27,560 its n, k, d and that sort of properties. 119 00:06:27,560 --> 00:06:32,510 So I want to take these two coordinates and make them 1, 120 00:06:32,510 --> 00:06:36,290 1, 1, 0, 0, 1, 1, 1. 121 00:06:36,290 --> 00:06:40,670 122 00:06:40,670 --> 00:06:43,210 Let's suppose that's a trellis-oriented generator 123 00:06:43,210 --> 00:06:44,260 matrix for the code. 124 00:06:44,260 --> 00:06:47,120 I think it is. 125 00:06:47,120 --> 00:06:52,940 It's got starting times here, here, here, and now here. 126 00:06:52,940 --> 00:06:59,570 And it's got ending times now here, here, here, and here. 127 00:06:59,570 --> 00:07:01,230 OK. 128 00:07:01,230 --> 00:07:06,020 Where I introduce the diverging and merging symbols 129 00:07:06,020 --> 00:07:10,270 that I used last time for starting and ending. 130 00:07:10,270 --> 00:07:11,880 All right, I think that is a 131 00:07:11,880 --> 00:07:14,080 trellis-oriented generator matrix. 132 00:07:14,080 --> 00:07:15,210 Well, it is. 133 00:07:15,210 --> 00:07:18,440 We found a matrix that has distinct starting times and 134 00:07:18,440 --> 00:07:19,690 ending times, so it is. 135 00:07:19,690 --> 00:07:23,810 136 00:07:23,810 --> 00:07:26,125 Now what's its state branch complexity? 137 00:07:26,125 --> 00:07:29,710 138 00:07:29,710 --> 00:07:35,320 The state dimension is 0. 139 00:07:35,320 --> 00:07:37,510 Now what are our spans now? 140 00:07:37,510 --> 00:07:40,730 141 00:07:40,730 --> 00:07:45,190 Two are the same, but two are larger. 142 00:07:45,190 --> 00:07:49,370 OK, so now when we count, we get that the state complexity 143 00:07:49,370 --> 00:07:54,200 goes 0, 1, 2, 3, there are actually four that are now 144 00:07:54,200 --> 00:07:55,830 active in the center. 145 00:07:55,830 --> 00:08:00,430 3, 2, 1, 0. 146 00:08:00,430 --> 00:08:03,090 OK, and that's worse than we had before, at 147 00:08:03,090 --> 00:08:05,670 least in one place. 148 00:08:05,670 --> 00:08:11,320 If we look at the dimensions of the branch spaces, which 149 00:08:11,320 --> 00:08:16,020 I'm arguing is more fundamental, then we get 1, 2, 150 00:08:16,020 --> 00:08:19,730 3, again, 4, 4. 151 00:08:19,730 --> 00:08:26,470 So the branch complexity is now 16 rather than 8. 152 00:08:26,470 --> 00:08:28,395 It's going to be more difficult, definitely, to 153 00:08:28,395 --> 00:08:30,620 decode this code with the Viterbi algorithm. 154 00:08:30,620 --> 00:08:33,669 This is a more complex representation of an 155 00:08:33,669 --> 00:08:37,429 effectively equivalent code. 156 00:08:37,429 --> 00:08:40,140 Well, so is there-- 157 00:08:40,140 --> 00:08:42,740 so this is a bad permutation, it's something I 158 00:08:42,740 --> 00:08:44,960 didn't want to do. 159 00:08:44,960 --> 00:08:48,570 Is it possible there's some good permutation that would 160 00:08:48,570 --> 00:08:51,410 reduce the complexity? 161 00:08:51,410 --> 00:08:56,350 In general, when we're given a block code, traditionally at 162 00:08:56,350 --> 00:08:59,726 least, we think of it as the coordinates as just being not 163 00:08:59,726 --> 00:09:00,560 in any particular order. 164 00:09:00,560 --> 00:09:06,310 We can write down a generator matrix, but the view is that 165 00:09:06,310 --> 00:09:11,960 any permutation of the symbol times is perfectly legitimate. 166 00:09:11,960 --> 00:09:14,990 Certainly, if we send it over a memoryless channel, it's not 167 00:09:14,990 --> 00:09:16,890 going to affect the performance of the code, what 168 00:09:16,890 --> 00:09:18,340 order we send the symbols in. 169 00:09:18,340 --> 00:09:24,830 So we should certainly regard all such codes as equivalent. 170 00:09:24,830 --> 00:09:27,210 But we can affect the complexity, for better or for 171 00:09:27,210 --> 00:09:31,330 worse, so we should try to -- 172 00:09:31,330 --> 00:09:34,670 presumably, if we want a more fundamental complexity metric 173 00:09:34,670 --> 00:09:38,480 for the code, we want to find the complexity of the least 174 00:09:38,480 --> 00:09:43,240 complex trellis realization under all permutations of the 175 00:09:43,240 --> 00:09:44,490 coordinates. 176 00:09:44,490 --> 00:09:47,210 177 00:09:47,210 --> 00:09:53,520 All right, so that's a good issue, but very little is 178 00:09:53,520 --> 00:09:56,295 known about this issue. 179 00:09:56,295 --> 00:10:00,240 It is known that finding the best permutation of an 180 00:10:00,240 --> 00:10:04,480 arbitrary linear code is an NP-hard problem. 181 00:10:04,480 --> 00:10:06,040 So this is going to be hard. 182 00:10:06,040 --> 00:10:08,790 183 00:10:08,790 --> 00:10:13,330 We know for some specific classes of codes what the best 184 00:10:13,330 --> 00:10:16,000 permutation is. 185 00:10:16,000 --> 00:10:20,220 For instance, the Reed-Muller codes are nice because it's 186 00:10:20,220 --> 00:10:23,810 been proved by Kasami and others that the standard 187 00:10:23,810 --> 00:10:26,270 coordinate ordering, the one that I've been using all along 188 00:10:26,270 --> 00:10:30,487 based on this length doubling construction, the u-u plus v 189 00:10:30,487 --> 00:10:35,030 construction, gives you the minimal trellis complexity. 190 00:10:35,030 --> 00:10:37,580 All right, so we know what the best ordering is for the 191 00:10:37,580 --> 00:10:38,830 Reed-Muller codes. 192 00:10:38,830 --> 00:10:42,580 193 00:10:42,580 --> 00:10:48,210 And in a minute, I'm going to prove what the best possible 194 00:10:48,210 --> 00:10:50,210 we can get bounds-- 195 00:10:50,210 --> 00:10:52,430 in particular, the Muder bounds, which I'm going to 196 00:10:52,430 --> 00:10:55,480 discuss next-- 197 00:10:55,480 --> 00:10:59,460 which bound the best possible complexity that 198 00:10:59,460 --> 00:11:01,040 you could ever get. 199 00:11:01,040 --> 00:11:04,350 And in some cases, we can find coordinate orderings that meet 200 00:11:04,350 --> 00:11:08,630 that bound, notably in the case of the 24-12-8 Golay 201 00:11:08,630 --> 00:11:12,180 code, we can find a coordinate ordering that's very closely 202 00:11:12,180 --> 00:11:15,295 related to the Reed-Muller ordering such that 203 00:11:15,295 --> 00:11:16,880 the bound is met. 204 00:11:16,880 --> 00:11:19,380 And so we know that we've found the best 205 00:11:19,380 --> 00:11:20,463 complexity for that code. 206 00:11:20,463 --> 00:11:22,590 It couldn't possibly be any better. 207 00:11:22,590 --> 00:11:27,200 But apart from certain very nice cases like those, we 208 00:11:27,200 --> 00:11:27,320 don't know. 209 00:11:27,320 --> 00:11:32,000 For instance, take an arbitrary BCH code. 210 00:11:32,000 --> 00:11:34,810 What's the best coordinate ordering for that? 211 00:11:34,810 --> 00:11:37,020 We don't know. 212 00:11:37,020 --> 00:11:43,880 So this is the open question in this subject, and it's now 213 00:11:43,880 --> 00:11:46,605 been open long enough and enough smart people have 214 00:11:46,605 --> 00:11:49,890 looked at it, and we've got this NP-hardness result, so 215 00:11:49,890 --> 00:11:53,070 that it looks like it's going to stay open indefinitely. 216 00:11:53,070 --> 00:11:56,150 217 00:11:56,150 --> 00:12:01,200 All right, let me discuss, however, this Muder bound, 218 00:12:01,200 --> 00:12:05,040 because it's simple and sometimes definitive. 219 00:12:05,040 --> 00:12:11,910 220 00:12:11,910 --> 00:12:12,165 Yes? 221 00:12:12,165 --> 00:12:13,415 AUDIENCE: [INAUDIBLE]? 222 00:12:13,415 --> 00:12:16,100 223 00:12:16,100 --> 00:12:21,520 PROFESSOR: Minimal branch complexity or any other 224 00:12:21,520 --> 00:12:22,540 measure that you like. 225 00:12:22,540 --> 00:12:25,192 They all are quite fungible, it turns out. 226 00:12:25,192 --> 00:12:31,310 227 00:12:31,310 --> 00:12:33,560 OK, The Muder bound. 228 00:12:33,560 --> 00:12:37,260 229 00:12:37,260 --> 00:12:44,925 Let's look at a particular code and suppose we just know 230 00:12:44,925 --> 00:12:46,980 its parameters, n, k, d. 231 00:12:46,980 --> 00:12:51,590 232 00:12:51,590 --> 00:12:56,750 Let's ask what the minimal trellis complexity could be 233 00:12:56,750 --> 00:13:01,740 for a particular coordinate ordering, say, that divides -- 234 00:13:01,740 --> 00:13:06,930 we have a certain number of past symbols, call it n_p, and 235 00:13:06,930 --> 00:13:11,690 a certain number of future symbols, n_f. 236 00:13:11,690 --> 00:13:15,740 And I'm going to ask what the minimum state complexity, or I 237 00:13:15,740 --> 00:13:17,370 could also ask what's the minimum 238 00:13:17,370 --> 00:13:21,170 branch complexity for-- 239 00:13:21,170 --> 00:13:24,220 let me do now a state complexity, going back to 240 00:13:24,220 --> 00:13:29,320 where we started from, for a a certain division in the past 241 00:13:29,320 --> 00:13:30,660 and future. 242 00:13:30,660 --> 00:13:38,825 To be concrete, let me take the 24-12-8 Golay code. 243 00:13:38,825 --> 00:13:43,570 It turns out that it's been proved that all codes with 244 00:13:43,570 --> 00:13:45,200 these parameters are equivalent. 245 00:13:45,200 --> 00:13:50,230 So we can just call this code the binary Golay code. 246 00:13:50,230 --> 00:13:57,240 And let's look for any division into 16 future 247 00:13:57,240 --> 00:14:00,420 coordinates and 8 past coordinates. 248 00:14:00,420 --> 00:14:02,980 So we don't know what the ordering is going to be, but 249 00:14:02,980 --> 00:14:04,790 we could say a few things. 250 00:14:04,790 --> 00:14:07,600 Remember how we get the state complexity. 251 00:14:07,600 --> 00:14:11,460 We look for what's the dimension of the past subcode. 252 00:14:11,460 --> 00:14:16,420 So here's a generator for the past subcode here, the code 253 00:14:16,420 --> 00:14:19,240 whose support is the past. 254 00:14:19,240 --> 00:14:23,370 We can ask what the dimension of the future subcode is, and 255 00:14:23,370 --> 00:14:28,035 then the dimension of the state space is the dimension 256 00:14:28,035 --> 00:14:30,100 of what's left. 257 00:14:30,100 --> 00:14:34,740 So it's dimension of c minus the dimension of the past 258 00:14:34,740 --> 00:14:40,380 subcode minus the dimension of the future subcode. 259 00:14:40,380 --> 00:14:43,220 State-space theorem. 260 00:14:43,220 --> 00:14:50,130 OK, but we know a few things about this code. 261 00:14:50,130 --> 00:14:53,440 We know what its length is and we know what its minimum 262 00:14:53,440 --> 00:14:54,990 distance is. 263 00:14:54,990 --> 00:15:00,340 So for this particular case, we have that c past is an 264 00:15:00,340 --> 00:15:08,450 8-something-8 code, because its minimum distance has got 265 00:15:08,450 --> 00:15:10,000 to be at least as great as this. 266 00:15:10,000 --> 00:15:13,670 It's a subcode of this code. 267 00:15:13,670 --> 00:15:18,370 So any non-zero code words in this past subcode have to have 268 00:15:18,370 --> 00:15:20,810 minimum weight of at least 8. 269 00:15:20,810 --> 00:15:22,660 Correct? 270 00:15:22,660 --> 00:15:24,530 All right. 271 00:15:24,530 --> 00:15:27,420 And similarly, the future subcode, in this 272 00:15:27,420 --> 00:15:30,590 case, is a 16-k -- 273 00:15:30,590 --> 00:15:32,100 let's call this past -- 274 00:15:32,100 --> 00:15:34,260 k-future-8 code. 275 00:15:34,260 --> 00:15:37,230 276 00:15:37,230 --> 00:15:39,720 At least 8. 277 00:15:39,720 --> 00:15:44,660 So that imposes some bounds on the dimension of this code. 278 00:15:44,660 --> 00:15:49,610 How large could the dimension possibly be? 279 00:15:49,610 --> 00:15:51,075 All right, anybody? 280 00:15:51,075 --> 00:15:58,210 For the past subcode its clearly, we have that k_p less 281 00:15:58,210 --> 00:15:59,240 than or equal to 1. 282 00:15:59,240 --> 00:16:01,530 We could have, at most, one generator here. 283 00:16:01,530 --> 00:16:03,660 And if there were one, it would have to be 284 00:16:03,660 --> 00:16:05,990 the all-1 code word. 285 00:16:05,990 --> 00:16:09,000 Or the all-1 symbols here, because it has 286 00:16:09,000 --> 00:16:11,000 to have weight 8. 287 00:16:11,000 --> 00:16:12,230 So that's all we can get. 288 00:16:12,230 --> 00:16:14,440 For the future code-- 289 00:16:14,440 --> 00:16:18,650 290 00:16:18,650 --> 00:16:21,560 well, I guess you don't know this, but you can guess that 291 00:16:21,560 --> 00:16:27,580 the Reed-Muller code with length 16 and minimum distance 292 00:16:27,580 --> 00:16:31,640 8 is the best there is, that distance 16, and that happens 293 00:16:31,640 --> 00:16:33,200 to be true. 294 00:16:33,200 --> 00:16:38,650 So k-future can't possibly be better than 5, which is the 295 00:16:38,650 --> 00:16:41,715 dimension of the Reed-Muller code with those parameters. 296 00:16:41,715 --> 00:16:46,680 297 00:16:46,680 --> 00:16:51,470 For more general cases, you have to look up tables on 298 00:16:51,470 --> 00:16:53,730 bounds on the best codes that have ever been found. 299 00:16:53,730 --> 00:16:57,860 But certainly for code lengths less than a 100, by this time 300 00:16:57,860 --> 00:17:02,230 we pretty much know all the best binary codes. 301 00:17:02,230 --> 00:17:06,089 So we can look up in a table what's the best code of length 302 00:17:06,089 --> 00:17:08,000 16 and distance 8? 303 00:17:08,000 --> 00:17:12,440 Well, it has five information symbols, dimension five. 304 00:17:12,440 --> 00:17:15,920 So we couldn't have more than one dimension here and we 305 00:17:15,920 --> 00:17:20,394 couldn't have more than five dimensions here, so the 306 00:17:20,394 --> 00:17:23,599 dimensional code itself is 12. 307 00:17:23,599 --> 00:17:26,119 We have to say that the dimension state-space is now 308 00:17:26,119 --> 00:17:28,160 going to be greater than equal to-- 309 00:17:28,160 --> 00:17:32,760 this is k_p minus 1 minus 5-- 310 00:17:32,760 --> 00:17:35,060 so, that's 6. 311 00:17:35,060 --> 00:17:41,810 So there have to be at least 6 generators down here that have 312 00:17:41,810 --> 00:17:45,000 support neither wholly on the past nor wholly on the future 313 00:17:45,000 --> 00:17:49,610 and therefore have to be included in the state-space. 314 00:17:49,610 --> 00:17:51,370 All right, that's the idea of the Muder bound. 315 00:17:51,370 --> 00:17:59,800 316 00:17:59,800 --> 00:18:03,390 Now, we could do this for every single state-space. 317 00:18:03,390 --> 00:18:06,960 We simply need to know what the dimension is of the best 318 00:18:06,960 --> 00:18:12,290 code of each length with minimum distance 8. 319 00:18:12,290 --> 00:18:17,520 And we can do a similar thing for every possible partition 320 00:18:17,520 --> 00:18:19,860 in the past and future. 321 00:18:19,860 --> 00:18:23,210 And that will give us a set of bounds, and we go through that 322 00:18:23,210 --> 00:18:26,616 exercise in the notes. 323 00:18:26,616 --> 00:18:32,670 And I can write it on the board, maybe for memory. 324 00:18:32,670 --> 00:18:37,950 And OK, that tells us what the best possible 325 00:18:37,950 --> 00:18:40,550 state profile is. 326 00:18:40,550 --> 00:18:55,510 Let me guess state dimension profile is 0, 1, 327 00:18:55,510 --> 00:19:00,706 2, 3, 4, 5, 6, 7. 328 00:19:00,706 --> 00:19:08,490 Then it goes down to 6, 7, 8, 9, then it goes down to 8. 329 00:19:08,490 --> 00:19:11,430 This is at the central state-space. 330 00:19:11,430 --> 00:19:17,060 This is for dividing it in the way that we just did. 331 00:19:17,060 --> 00:19:20,060 Let's do this here. 332 00:19:20,060 --> 00:19:27,870 For the central state-space, what's the best k_p for a 12, 333 00:19:27,870 --> 00:19:31,760 k_p, 8 code? 334 00:19:31,760 --> 00:19:33,790 Let's make the past and future both 12. 335 00:19:33,790 --> 00:19:38,470 336 00:19:38,470 --> 00:19:40,620 Anyone want to guess what the best dimension is? 337 00:19:40,620 --> 00:19:43,530 338 00:19:43,530 --> 00:19:45,280 Let's think about it. 339 00:19:45,280 --> 00:19:48,520 Let's think of a code that's going to have to have 340 00:19:48,520 --> 00:19:49,620 minimum weight 8. 341 00:19:49,620 --> 00:19:54,456 So let's start out with g1. 342 00:19:54,456 --> 00:20:00,140 And we'll just make that look like that. 343 00:20:00,140 --> 00:20:01,250 It's going to have length 12. 344 00:20:01,250 --> 00:20:04,910 It's going to have to have at least eight 1s, so let's make 345 00:20:04,910 --> 00:20:07,030 them like that. 346 00:20:07,030 --> 00:20:12,704 We want to add another 1 so that the total code has a 347 00:20:12,704 --> 00:20:15,350 minimum distance 8. 348 00:20:15,350 --> 00:20:18,250 Subject to up to permutations, there's only one 349 00:20:18,250 --> 00:20:18,865 thing it can be. 350 00:20:18,865 --> 00:20:23,080 We need another weight-8 generator, and it needs to 351 00:20:23,080 --> 00:20:26,150 differ from this in at least four places so that when we 352 00:20:26,150 --> 00:20:30,840 take the mod 2 sum of these two, it has distance 8. 353 00:20:30,840 --> 00:20:35,140 And we want g1 plus g2. 354 00:20:35,140 --> 00:20:38,600 This is really the only way it can be. 355 00:20:38,600 --> 00:20:39,900 Are you with me? 356 00:20:39,900 --> 00:20:44,030 We have four code words in this dimension 2 code. 357 00:20:44,030 --> 00:20:45,670 And if they're going to have minimum distance 8, they're 358 00:20:45,670 --> 00:20:47,460 going to have to look like this, subject to up to 359 00:20:47,460 --> 00:20:48,710 permutation. 360 00:20:48,710 --> 00:20:50,540 361 00:20:50,540 --> 00:20:54,010 So from that little argument, we conclude that, in this 362 00:20:54,010 --> 00:20:56,410 case, this is less than or equal to 2. 363 00:20:56,410 --> 00:21:01,470 This is less than or equal to 2, the Muder bound becomes 364 00:21:01,470 --> 00:21:05,865 that the dimension of the state-space has 365 00:21:05,865 --> 00:21:07,115 to be at least 8. 366 00:21:07,115 --> 00:21:10,880 367 00:21:10,880 --> 00:21:13,640 And so in the same way, and then it becomes symmetrical on 368 00:21:13,640 --> 00:21:14,370 the other side. 369 00:21:14,370 --> 00:21:17,410 So I'm pretty sure I get it right. 370 00:21:17,410 --> 00:21:20,800 And from memory, that's the best possible state dimension 371 00:21:20,800 --> 00:21:25,140 profile, dot dot dot. 372 00:21:25,140 --> 00:21:29,505 And we actually know a coordinate ordering for the 373 00:21:29,505 --> 00:21:33,500 Golay code, a generator matrix for the Golay code, that gives 374 00:21:33,500 --> 00:21:35,440 this state dimension profile. 375 00:21:35,440 --> 00:21:39,070 So from that, we can conclude that we know an optimum 376 00:21:39,070 --> 00:21:44,670 coordinate ordering for the Golay code. 377 00:21:44,670 --> 00:21:47,140 I just want you to get the essence of the argument here. 378 00:21:47,140 --> 00:21:49,670 379 00:21:49,670 --> 00:21:50,920 You with me? 380 00:21:50,920 --> 00:21:52,660 381 00:21:52,660 --> 00:21:56,270 For branch spaces, the argument is much the same. 382 00:21:56,270 --> 00:22:02,220 Remember in branch spaces, we basically -- 383 00:22:02,220 --> 00:22:05,790 let me say it in a slightly different way than we did 384 00:22:05,790 --> 00:22:10,460 before -- we take the past to be up to time k. 385 00:22:10,460 --> 00:22:13,460 And we have the symbol time, which is just one time in here 386 00:22:13,460 --> 00:22:15,900 for the actual symbol we're looking at. 387 00:22:15,900 --> 00:22:18,770 And the future is one unit shorter than it was for the 388 00:22:18,770 --> 00:22:20,070 state-space. 389 00:22:20,070 --> 00:22:24,380 So when we compute the branch dimension, the 390 00:22:24,380 --> 00:22:28,390 past goes up to k. 391 00:22:28,390 --> 00:22:32,570 So this is 0 and before k. 392 00:22:32,570 --> 00:22:36,780 And the future goes from k plus one up to n. 393 00:22:36,780 --> 00:22:40,410 394 00:22:40,410 --> 00:22:44,690 So this is what the matrix looks like for this. 395 00:22:44,690 --> 00:22:51,030 And the effect over here is that the future subcode now is 396 00:22:51,030 --> 00:22:52,900 one unit shorter. 397 00:22:52,900 --> 00:22:55,050 So let's do the same calculation here. 398 00:22:55,050 --> 00:22:58,720 399 00:22:58,720 --> 00:23:04,830 If we look at a time where we have 12 in the past, this one 400 00:23:04,830 --> 00:23:07,940 time, the 13th symbol time that we're actually looking at 401 00:23:07,940 --> 00:23:14,370 and then 11 more in the future, the past subcode, 402 00:23:14,370 --> 00:23:18,160 again, could have dimension up to 2 because it has length 12 403 00:23:18,160 --> 00:23:19,950 and minimum distance 8. 404 00:23:19,950 --> 00:23:25,500 The future subcode now has a shorter length and by this 405 00:23:25,500 --> 00:23:27,395 little argument I went through here. 406 00:23:27,395 --> 00:23:30,740 If we take out one of these coordinates, it's 407 00:23:30,740 --> 00:23:32,790 not going to work. 408 00:23:32,790 --> 00:23:37,210 So if we only have length 11, the future subcode is the 409 00:23:37,210 --> 00:23:38,380 dimension -- 410 00:23:38,380 --> 00:23:39,530 still can only be 1. 411 00:23:39,530 --> 00:23:43,890 We can't possibly add another second generator in there. 412 00:23:43,890 --> 00:23:51,790 And so the dimension of the branch space has to be vastly 413 00:23:51,790 --> 00:23:57,170 greater than or equal to 9, at this point. 414 00:23:57,170 --> 00:24:00,760 OK, so the Golay code has branch complexity of at least 415 00:24:00,760 --> 00:24:04,760 512, we conclude. 416 00:24:04,760 --> 00:24:10,060 And, in fact, this best possible branch dimension 417 00:24:10,060 --> 00:24:12,880 profile looks like this. 418 00:24:12,880 --> 00:24:15,240 The branches are in the middle here. 419 00:24:15,240 --> 00:24:19,800 They're 1, 2, 3, 4, 5, 6, 6. 420 00:24:19,800 --> 00:24:25,400 Basically we have generators which look like this. 421 00:24:25,400 --> 00:24:28,010 And then there's one that comes down, there are three 422 00:24:28,010 --> 00:24:31,090 more that goes up and one that comes down. 423 00:24:31,090 --> 00:24:41,125 And from that, we get 1, 2, 3, 4, 6, 7, 7, 8 -- 424 00:24:41,125 --> 00:24:44,820 425 00:24:44,820 --> 00:24:48,290 that can't be right. 426 00:24:48,290 --> 00:24:53,290 Goes down 7, goes up, it's still 7, then it goes up to 8, 427 00:24:53,290 --> 00:25:00,330 9, and 9, 9, 9. 428 00:25:00,330 --> 00:25:01,660 Something like that. 429 00:25:01,660 --> 00:25:03,685 That one's probably wrong, and symmetrically. 430 00:25:03,685 --> 00:25:06,470 431 00:25:06,470 --> 00:25:08,565 Where this 9 is the one we just calculated. 432 00:25:08,565 --> 00:25:11,190 433 00:25:11,190 --> 00:25:15,850 Similarly, if you come down to the neighbor of this 6, it's 434 00:25:15,850 --> 00:25:18,320 got a merge on one side, a diverge on the other 435 00:25:18,320 --> 00:25:20,870 side, and that's 7. 436 00:25:20,870 --> 00:25:26,450 OK, so we conclude that the Golay code has branch 437 00:25:26,450 --> 00:25:29,090 dimension not less than 512 within 9. 438 00:25:29,090 --> 00:25:29,579 Yeah? 439 00:25:29,579 --> 00:25:36,106 AUDIENCE: Suppose we take past and future separated by 12, 12 440 00:25:36,106 --> 00:25:38,830 in the past, 12 in the future. 441 00:25:38,830 --> 00:25:41,620 Suppose we did it as they did in [UNINTELLIGIBLE] state? 442 00:25:41,620 --> 00:25:43,250 PROFESSOR: OK, but that's not the way we do branches. 443 00:25:43,250 --> 00:25:45,290 We've got to isolate the symbol time. 444 00:25:45,290 --> 00:25:49,030 AUDIENCE: Suppose we took 12-branch-12 and we calculate 445 00:25:49,030 --> 00:25:52,450 the state complexity, that's 8. 446 00:25:52,450 --> 00:25:59,435 And then we took 13 and 11, we get the state complexity 9. 447 00:25:59,435 --> 00:26:03,530 448 00:26:03,530 --> 00:26:10,130 PROFESSOR: All right, you're saying if we do this, we 449 00:26:10,130 --> 00:26:12,670 should get the state complexity 9. 450 00:26:12,670 --> 00:26:14,840 And that's correct, we do. 451 00:26:14,840 --> 00:26:18,095 AUDIENCE: The maximum of those two numbers, is it the same as 452 00:26:18,095 --> 00:26:19,025 branch complexity? 453 00:26:19,025 --> 00:26:20,902 PROFESSOR: It turns out to be in this case. 454 00:26:20,902 --> 00:26:21,630 AUDIENCE: Not always? 455 00:26:21,630 --> 00:26:22,880 PROFESSOR: Not always. 456 00:26:22,880 --> 00:26:28,290 457 00:26:28,290 --> 00:26:34,070 But if we go to 14, 10, then it turns out we can add 458 00:26:34,070 --> 00:26:39,370 another one here because we can add another generator that 459 00:26:39,370 --> 00:26:43,020 looks like something like this. 460 00:26:43,020 --> 00:26:50,660 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1. 461 00:26:50,660 --> 00:26:53,860 So if we have two more units, we can add another generator. 462 00:26:53,860 --> 00:26:56,380 Sort of Reed-Muller style. 463 00:26:56,380 --> 00:27:02,886 And so this goes up to 3 and this goes down to 8. 464 00:27:02,886 --> 00:27:05,460 AUDIENCE: So the branch complexity has to be at least 465 00:27:05,460 --> 00:27:07,020 the maximum of those two, right? 466 00:27:07,020 --> 00:27:08,270 PROFESSOR: Obviously, right. 467 00:27:08,270 --> 00:27:12,085 468 00:27:12,085 --> 00:27:14,830 And really, together these give you the story of where 469 00:27:14,830 --> 00:27:17,820 the thing has to diverge and merge. 470 00:27:17,820 --> 00:27:20,720 In this case, this is a self-dual code, so you only do 471 00:27:20,720 --> 00:27:22,170 one at each time. 472 00:27:22,170 --> 00:27:24,090 So with this sort of thing, you can 473 00:27:24,090 --> 00:27:26,800 pretty well pin it down. 474 00:27:26,800 --> 00:27:31,500 Again, I don't know how far to take this lecture. 475 00:27:31,500 --> 00:27:33,900 I'm really just trying to get to you the idea of the Muder 476 00:27:33,900 --> 00:27:37,730 bound from this kind of argument, therefore we can get 477 00:27:37,730 --> 00:27:40,980 a bound on the best possible state dimension profile and 478 00:27:40,980 --> 00:27:45,080 the best possible branch complexity profile. 479 00:27:45,080 --> 00:27:52,840 If you compute this bound and you're able to find an actual 480 00:27:52,840 --> 00:27:55,000 generator matrix that meets this bound, you know you've 481 00:27:55,000 --> 00:27:56,990 found the best one. 482 00:27:56,990 --> 00:28:01,770 But this rarely happens, this is a very special code. 483 00:28:01,770 --> 00:28:05,230 It's not the way you prove the optimality of the longer 484 00:28:05,230 --> 00:28:07,990 Reed-Muller codes, they're not good enough to -- 485 00:28:07,990 --> 00:28:12,830 this really assumes that you can get the best possible 486 00:28:12,830 --> 00:28:15,190 sub-codes at every point. 487 00:28:15,190 --> 00:28:17,710 And you can imagine that's very hard to do. 488 00:28:17,710 --> 00:28:20,470 It would be a very special case where you can optimize 489 00:28:20,470 --> 00:28:24,655 this for every single possible cut between past and future. 490 00:28:24,655 --> 00:28:27,520 491 00:28:27,520 --> 00:28:31,760 OK, so that's the Muder bound. 492 00:28:31,760 --> 00:28:37,460 And you have a homework exercise on that as well. 493 00:28:37,460 --> 00:28:41,900 Oh, there's another case where we can find what the best 494 00:28:41,900 --> 00:28:43,500 possible trellis complexity is. 495 00:28:43,500 --> 00:28:49,900 It's MDS codes over GFq, and you're going to do that on the 496 00:28:49,900 --> 00:28:51,580 homework as well. 497 00:28:51,580 --> 00:28:59,030 And there, we basically show that an MDS code, its n, k, d 498 00:28:59,030 --> 00:29:02,280 are such that it has to have the worst possible state 499 00:29:02,280 --> 00:29:06,470 complexity and branch complexity. 500 00:29:06,470 --> 00:29:09,960 So because it's such a tightly-packed code, in 501 00:29:09,960 --> 00:29:15,260 Hamming distance sense, it forces you to have the worst 502 00:29:15,260 --> 00:29:18,155 possible complexity that you could have. 503 00:29:18,155 --> 00:29:22,530 In other words, you have diverges immediately and as 504 00:29:22,530 --> 00:29:27,280 long as you can, and then they're all packed together on 505 00:29:27,280 --> 00:29:29,920 the left side and the merges are all packed together on the 506 00:29:29,920 --> 00:29:33,380 right side, and that's the only way it can be. 507 00:29:33,380 --> 00:29:35,675 I'm just speaking very impressionistically. 508 00:29:35,675 --> 00:29:37,865 But you'll see what I mean when you get to the homework, 509 00:29:37,865 --> 00:29:40,610 if you haven't done it already. 510 00:29:40,610 --> 00:29:48,970 OK, so we also know MDS codes, we know their trellis 511 00:29:48,970 --> 00:29:51,390 complexity and it's terrible. 512 00:29:51,390 --> 00:29:55,700 We can't get any effective reduction. 513 00:29:55,700 --> 00:29:58,860 Really, over exhaustive decoding methods with MDS 514 00:29:58,860 --> 00:30:03,040 codes, we find that the trellis complexity is like 515 00:30:03,040 --> 00:30:05,360 either q to the k or q to the n minus k, 516 00:30:05,360 --> 00:30:06,720 depending on which is less. 517 00:30:06,720 --> 00:30:10,959 518 00:30:10,959 --> 00:30:13,205 So that's as bad as it could possibly be. 519 00:30:13,205 --> 00:30:19,790 So if the trellis doesn't help for MDS codes, in general 520 00:30:19,790 --> 00:30:20,930 cyclic codes -- 521 00:30:20,930 --> 00:30:23,940 which were subject to a lot of study -- if we put the code 522 00:30:23,940 --> 00:30:28,790 into cyclic form, which if you remember Reed-Solomon codes 523 00:30:28,790 --> 00:30:32,890 and BCH codes can always be put into cyclic form, at least 524 00:30:32,890 --> 00:30:35,050 the shortened versions of them. 525 00:30:35,050 --> 00:30:37,610 In that case, the generator matrix looks 526 00:30:37,610 --> 00:30:39,970 like this one up here. 527 00:30:39,970 --> 00:30:47,880 It looks like x, x, x, x, x, x, x, 0, 0, 0, 0, 0, 0, 0, x, 528 00:30:47,880 --> 00:30:53,820 x, x, x, x, x, x, 0, 0, x, x, x, x, x, x, x. 529 00:30:53,820 --> 00:30:56,830 In other words, the generators are just cyclic 530 00:30:56,830 --> 00:30:58,080 shifts of one another. 531 00:30:58,080 --> 00:31:00,500 532 00:31:00,500 --> 00:31:07,300 Down to x, x, x, x, x, x, x, 0, 0, 0, 0. 533 00:31:07,300 --> 00:31:10,570 534 00:31:10,570 --> 00:31:13,190 That is a generator matrix that is a trellis-oriented 535 00:31:13,190 --> 00:31:15,930 generator matrix because all the starting times are 536 00:31:15,930 --> 00:31:18,950 different and all the stopping times are different. 537 00:31:18,950 --> 00:31:22,080 So this is the trellis-oriented generator 538 00:31:22,080 --> 00:31:27,960 matrix for cyclic code, and it's in the coordinate 539 00:31:27,960 --> 00:31:30,440 ordering in which it is cyclic. 540 00:31:30,440 --> 00:31:36,210 And it's always going to have this worst case -- all the 541 00:31:36,210 --> 00:31:40,170 branch diverges are going to occur over here and you may 542 00:31:40,170 --> 00:31:41,390 get nothing for a while. 543 00:31:41,390 --> 00:31:47,330 And then all the merges over here, which if it's a rate 544 00:31:47,330 --> 00:31:51,230 less than 1/2 code, this means that the trellis complexity is 545 00:31:51,230 --> 00:31:55,070 basically going to be 2 to the k. 546 00:31:55,070 --> 00:31:57,320 Or if it's a high-rate code, it means it's going to be 2 to 547 00:31:57,320 --> 00:31:58,710 the n minus k. 548 00:31:58,710 --> 00:32:01,180 And again, you get no improvement over some 549 00:32:01,180 --> 00:32:04,410 exhaustive minimum distance decoding, or maximum 550 00:32:04,410 --> 00:32:06,770 likelihood decoding. 551 00:32:06,770 --> 00:32:10,240 So cyclic is very bad from a trellis point of view. 552 00:32:10,240 --> 00:32:16,360 553 00:32:16,360 --> 00:32:19,110 So I think that's about all the general 554 00:32:19,110 --> 00:32:22,010 comments I can make. 555 00:32:22,010 --> 00:32:27,060 The very last thing that I want to mention is 556 00:32:27,060 --> 00:32:29,580 we now want to -- 557 00:32:29,580 --> 00:32:33,580 all right, so through this nice little theory, we've got 558 00:32:33,580 --> 00:32:40,480 a way of trellis decoding (maximum likelihood decoding), 559 00:32:40,480 --> 00:32:46,400 via the Viterbi algorithm, of block codes. 560 00:32:46,400 --> 00:32:52,600 And now, did we achieve any improvements in terms of 561 00:32:52,600 --> 00:33:00,360 performance versus complexity versus the convolutional codes 562 00:33:00,360 --> 00:33:03,390 that we looked at previously? 563 00:33:03,390 --> 00:33:06,120 564 00:33:06,120 --> 00:33:12,040 And the basic answer is, no. 565 00:33:12,040 --> 00:33:17,850 If you compare the tables, 566 00:33:17,850 --> 00:33:19,970 convolutional are always better. 567 00:33:19,970 --> 00:33:22,590 I'll just say it. 568 00:33:22,590 --> 00:33:26,890 You can imagine that there have been long debates in the 569 00:33:26,890 --> 00:33:31,580 fraternity, and that certainly some of the motivation for 570 00:33:31,580 --> 00:33:34,390 trellis decoding of block codes was, oh, gee, now we 571 00:33:34,390 --> 00:33:37,710 have a good way to decode block codes, just like we did 572 00:33:37,710 --> 00:33:39,500 for convolutional codes. 573 00:33:39,500 --> 00:33:43,500 And maybe now we'll find that block codes are worthwhile. 574 00:33:43,500 --> 00:33:46,190 But the answer is they're not. 575 00:33:46,190 --> 00:33:50,840 I should put this comparison into Chapter 10. 576 00:33:50,840 --> 00:33:54,205 You can actually get this comparison by taking the table 577 00:33:54,205 --> 00:33:56,070 -- for instance, for Reed-Muller codes. 578 00:33:56,070 --> 00:33:59,010 In terms of performance versus complexity, Reed-Muller codes 579 00:33:59,010 --> 00:34:00,250 are very good. 580 00:34:00,250 --> 00:34:03,040 I would say as a general statement, they're the best 581 00:34:03,040 --> 00:34:07,470 block codes that we know of, in terms of performance versus 582 00:34:07,470 --> 00:34:11,679 trellis decoding complexity, because they have this nice 583 00:34:11,679 --> 00:34:16,080 structure that gives them very low-complexity trellises for 584 00:34:16,080 --> 00:34:19,820 their n, k, d parameters. 585 00:34:19,820 --> 00:34:22,909 So in this sense, it's a fair comparison. 586 00:34:22,909 --> 00:34:25,540 Reed-Muller codes are the best general class we know of in 587 00:34:25,540 --> 00:34:27,810 terms of trellis complexity. 588 00:34:27,810 --> 00:34:35,270 We compare them against binary convolutional codes and 589 00:34:35,270 --> 00:34:38,530 they're not nearly as good. 590 00:34:38,530 --> 00:34:40,330 I've already talked about the comparison. 591 00:34:40,330 --> 00:34:44,630 It seems like a very fair comparison to take our rate 592 00:34:44,630 --> 00:34:48,310 1/2 four-state convolutional code example, which had 593 00:34:48,310 --> 00:34:53,690 distance 5, which had not only a nominal coding gain of 4 dB, 594 00:34:53,690 --> 00:34:58,020 but it also has an effective coding gain of 4 dB. 595 00:34:58,020 --> 00:35:03,470 And for a block code, this 8, 4, 4 code is a very fair 596 00:35:03,470 --> 00:35:04,810 comparison. 597 00:35:04,810 --> 00:35:08,780 It's also rate 1/2, so it has the same spectral efficiency. 598 00:35:08,780 --> 00:35:12,770 It also has a four-state trellis that looks very much 599 00:35:12,770 --> 00:35:17,100 like a rate 1/2 convolutional code trellis. 600 00:35:17,100 --> 00:35:20,265 But it turns out it only has minimum distance of 4 rather 601 00:35:20,265 --> 00:35:27,720 than 5, so its nominal coding gain is only 3 dB 602 00:35:27,720 --> 00:35:28,850 rather than 4 dB. 603 00:35:28,850 --> 00:35:32,250 Its effective coding gain is less. 604 00:35:32,250 --> 00:35:37,400 Its effective coding gain is 2.6 dN because it has 14 605 00:35:37,400 --> 00:35:41,030 nearest neighbors, which is a 3 1/2 nearest neighbors per 606 00:35:41,030 --> 00:35:45,550 bit, and that costs you about 4/10 of a dN. 607 00:35:45,550 --> 00:35:50,340 And so really, you compare the performance of the 608 00:35:50,340 --> 00:35:53,040 convolutional to the block code, that's very much better. 609 00:35:53,040 --> 00:35:56,620 610 00:35:56,620 --> 00:36:00,330 Another head-to-head comparison might be between 611 00:36:00,330 --> 00:36:04,380 the 32, 16, 8 Reed-Muller code. 612 00:36:04,380 --> 00:36:06,680 That's a rate 1/2 code. 613 00:36:06,680 --> 00:36:12,030 It's a very good one, it's known to be the best with 614 00:36:12,030 --> 00:36:14,460 those parameters, n, k, d. 615 00:36:14,460 --> 00:36:16,670 So what's its nominal coding gain? 616 00:36:16,670 --> 00:36:17,070 Anyone? 617 00:36:17,070 --> 00:36:18,860 It has rate 1/2, minimum distance 8, its 618 00:36:18,860 --> 00:36:22,410 nominal coding gain is? 619 00:36:22,410 --> 00:36:24,570 Hello? 620 00:36:24,570 --> 00:36:26,460 6 dB. 621 00:36:26,460 --> 00:36:29,100 You take the rate times the minimum distance-- 622 00:36:29,100 --> 00:36:31,210 1/2 times 8, you get 4. 623 00:36:31,210 --> 00:36:32,890 So that's the nominal coding gain. 624 00:36:32,890 --> 00:36:38,030 A factor of 4, which in dB terms is 6 dB. 625 00:36:38,030 --> 00:36:41,510 But again, it has lots of minimum-weight code words. 626 00:36:41,510 --> 00:36:45,850 It has 620 minimum-weight code words. 627 00:36:45,850 --> 00:36:48,760 And therefore, the number of minimum-weight code words per 628 00:36:48,760 --> 00:36:54,690 bit is 39, which is between five and six factors of 3, so 629 00:36:54,690 --> 00:37:01,220 it costs you 5 and 1/2 times 0.2 dB in terms of effective 630 00:37:01,220 --> 00:37:03,100 coding gain, which brings you down to an effective coding 631 00:37:03,100 --> 00:37:04,350 gain of 4.9 dB. 632 00:37:04,350 --> 00:37:06,700 633 00:37:06,700 --> 00:37:08,142 I assume you're all very fluent in 634 00:37:08,142 --> 00:37:11,970 doing these, as I am. 635 00:37:11,970 --> 00:37:15,420 And what's its complexity? 636 00:37:15,420 --> 00:37:21,770 If you look its state complexity, let's take-- 637 00:37:21,770 --> 00:37:24,320 we're looking at the 32, 16, 8. 638 00:37:24,320 --> 00:37:29,960 And all the Reed-Muller codes have four section trellises, 639 00:37:29,960 --> 00:37:32,040 you proved that this week. 640 00:37:32,040 --> 00:37:36,350 If you just measure the state complexity of four places, 641 00:37:36,350 --> 00:37:40,130 equally spaced with four equal sections, then all the 642 00:37:40,130 --> 00:37:44,800 trellises wind up looking like this, for a general 643 00:37:44,800 --> 00:37:47,390 Reed-Muller trellis -- 644 00:37:47,390 --> 00:37:48,640 a four-section trellis. 645 00:37:48,640 --> 00:37:53,400 646 00:37:53,400 --> 00:37:57,250 And for the 32, 16, 8, it turns out you get eight 647 00:37:57,250 --> 00:38:02,570 parallel sections like that with eight of these each. 648 00:38:02,570 --> 00:38:05,875 In any case, it's 64 states at each of these boundaries. 649 00:38:05,875 --> 00:38:09,830 650 00:38:09,830 --> 00:38:12,600 So you could say it's a 64-state trellis. 651 00:38:12,600 --> 00:38:16,370 But if you're more honest, just exactly as with the Golay 652 00:38:16,370 --> 00:38:20,890 code calculation we went through, you find that the 653 00:38:20,890 --> 00:38:25,480 maximum branch complexity is 512. 654 00:38:25,480 --> 00:38:36,710 So branch complexity is 512. 655 00:38:36,710 --> 00:38:43,440 OK, and its nominal coding gain we said is 6 dB and its 656 00:38:43,440 --> 00:38:47,450 effective coding gain is 4.9 dB. 657 00:38:47,450 --> 00:38:51,150 658 00:38:51,150 --> 00:38:56,065 OK, so let's look for a convolutional code of rate 1/2 659 00:38:56,065 --> 00:38:59,620 that has a coding gain of about 5 or 6 dB. 660 00:38:59,620 --> 00:39:02,320 661 00:39:02,320 --> 00:39:05,010 And how far do we have to go? 662 00:39:05,010 --> 00:39:08,625 We actually only have to go up to a-- 663 00:39:08,625 --> 00:39:12,010 well, a 32-state code has an effective 664 00:39:12,010 --> 00:39:13,520 coding gain of 5.6 dB. 665 00:39:13,520 --> 00:39:19,300 A 64-state code we can get up to an effective coding gain of 666 00:39:19,300 --> 00:39:22,060 about 6 dB. 667 00:39:22,060 --> 00:39:26,700 In fact, the 64-state rate 1/2 convolutional code was a 668 00:39:26,700 --> 00:39:31,680 standard for a very long time, first in space communications 669 00:39:31,680 --> 00:39:33,810 and then in much more general applications. 670 00:39:33,810 --> 00:39:37,650 This was the code that Linkabit Corporation -- 671 00:39:37,650 --> 00:39:39,280 Jacobs and Viterbi -- 672 00:39:39,280 --> 00:39:43,730 rode to fame and fortune, made their first pile before they 673 00:39:43,730 --> 00:39:47,520 went on to found Qualcomm, where they made 674 00:39:47,520 --> 00:39:50,850 a much bigger pile. 675 00:39:50,850 --> 00:39:53,530 And so this was their-- 676 00:39:53,530 --> 00:39:55,936 they call it k equals 7. 677 00:39:55,936 --> 00:40:01,120 But in my terms, nu equals 6, a 64-state rate 1/2 678 00:40:01,120 --> 00:40:02,170 convolutional code. 679 00:40:02,170 --> 00:40:12,170 So let's take that rate 1/2 64-state convolutional code, 680 00:40:12,170 --> 00:40:14,500 just with the standard trellis. 681 00:40:14,500 --> 00:40:23,945 So its branch complexity is 128 at every time. 682 00:40:23,945 --> 00:40:27,920 It has a more regular structure than this guy. 683 00:40:27,920 --> 00:40:33,860 Its nominal coding gain is -- 684 00:40:33,860 --> 00:40:37,040 there are actually two of them, and I don't remember. 685 00:40:37,040 --> 00:40:40,210 There's one of them that has a nominal coding gain of 7, the 686 00:40:40,210 --> 00:40:41,030 other one is 6.5. 687 00:40:41,030 --> 00:40:45,210 So it's at least 6.5 dB. 688 00:40:45,210 --> 00:40:51,030 And its effective coding gain is still 6.1 dB and it's 689 00:40:51,030 --> 00:40:52,390 clearly much better than this. 690 00:40:52,390 --> 00:40:57,900 691 00:40:57,900 --> 00:41:03,650 If you kind of try to plot a graph of this, you'll see that 692 00:41:03,650 --> 00:41:07,220 the effective coding gains of convolutional codes with the 693 00:41:07,220 --> 00:41:11,950 same state complexity is at least one dB better, as a 694 00:41:11,950 --> 00:41:14,910 general trend, than that of block codes. 695 00:41:14,910 --> 00:41:21,180 And of course, if you do it in terms of branch complexity, 696 00:41:21,180 --> 00:41:24,160 it's an even better comparison. 697 00:41:24,160 --> 00:41:31,480 So what happened was there was about a decade of effort on 698 00:41:31,480 --> 00:41:35,150 basically finding good trellis representations of block 699 00:41:35,150 --> 00:41:38,690 codes, and the net of it was that they couldn't get 700 00:41:38,690 --> 00:41:42,100 anywhere close to finding anything as good as 701 00:41:42,100 --> 00:41:43,100 convolutional codes. 702 00:41:43,100 --> 00:41:46,910 Although, it's a nice little piece of theory. 703 00:41:46,910 --> 00:41:49,460 And sometimes you want to block codes for other reasons. 704 00:41:49,460 --> 00:41:52,490 Sometimes you want short blocks. 705 00:41:52,490 --> 00:41:57,100 Sometimes you're limited to a block of less than 100. 706 00:41:57,100 --> 00:42:00,040 And in that case, you're certainly not going to do 707 00:42:00,040 --> 00:42:03,160 better in terms of n, k, d than the best block code of a 708 00:42:03,160 --> 00:42:04,770 length 100. 709 00:42:04,770 --> 00:42:06,700 Why are convolutional codes better? 710 00:42:06,700 --> 00:42:11,610 First of all, they obviously naturally are well-adapted to 711 00:42:11,610 --> 00:42:12,840 the trellis idea. 712 00:42:12,840 --> 00:42:20,300 They are linear time invariant machines that just have very 713 00:42:20,300 --> 00:42:22,550 nice, regular trellises. 714 00:42:22,550 --> 00:42:27,170 So they're naturally adapted. 715 00:42:27,170 --> 00:42:31,740 But also, they have exactly the right-- 716 00:42:31,740 --> 00:42:35,580 a rate 1/2 code is diverge, merge, diverge, merge, 717 00:42:35,580 --> 00:42:38,630 diverge, merge forever. 718 00:42:38,630 --> 00:42:41,270 If you want to make it into a block code, you can terminate 719 00:42:41,270 --> 00:42:43,980 it or you can tail-bite it, which is something we're going 720 00:42:43,980 --> 00:42:46,970 to talk about shortly. 721 00:42:46,970 --> 00:42:50,150 But whichever way you do it, to make it into a block code, 722 00:42:50,150 --> 00:42:50,780 you lose something. 723 00:42:50,780 --> 00:42:54,040 The termination, you don't lose minimum distance, but you 724 00:42:54,040 --> 00:42:54,820 do lose rate. 725 00:42:54,820 --> 00:42:58,090 You have to put in some dummy information symbols, so the 726 00:42:58,090 --> 00:43:02,250 rate becomes less than 1/2 if you terminate this code. 727 00:43:02,250 --> 00:43:05,360 Nonetheless, as I believe I commented earlier, that's a 728 00:43:05,360 --> 00:43:09,620 good way to come up with good block codes, is to terminate a 729 00:43:09,620 --> 00:43:12,140 good convolutional code. 730 00:43:12,140 --> 00:43:14,270 If you choose the parameters right, you can come up with 731 00:43:14,270 --> 00:43:16,350 pretty good codes. 732 00:43:16,350 --> 00:43:19,660 But nonetheless, in terms of block code parameters, you're 733 00:43:19,660 --> 00:43:23,650 not going to do better than what 50 years of block code, 734 00:43:23,650 --> 00:43:25,580 algebraic coding theory has come up with. 735 00:43:25,580 --> 00:43:30,240 So for short block codes with length less than 100, really 736 00:43:30,240 --> 00:43:34,100 the best we know of right now-- let's say 128-- 737 00:43:34,100 --> 00:43:37,890 is to pick a good Reed-Muller code or BCH code, find its 738 00:43:37,890 --> 00:43:41,640 best trellis and do maximum likelihood decoding. 739 00:43:41,640 --> 00:43:46,010 740 00:43:46,010 --> 00:43:49,110 It used to be that you thought of 64 states as a lot of 741 00:43:49,110 --> 00:43:54,740 states, but clearly, technology has progressed. 742 00:43:54,740 --> 00:43:59,480 And now you wouldn't blink at 1,000 states or maybe even 743 00:43:59,480 --> 00:44:02,065 8,000 or 16,000 states. 744 00:44:02,065 --> 00:44:07,205 The biggest Viterbi decoder that was ever built had 2 to 745 00:44:07,205 --> 00:44:10,860 the 14 states for a space program out of JPL. 746 00:44:10,860 --> 00:44:14,970 747 00:44:14,970 --> 00:44:17,460 So anyway, it's doable now. 748 00:44:17,460 --> 00:44:22,860 749 00:44:22,860 --> 00:44:26,690 If you needed the best possible performance and you 750 00:44:26,690 --> 00:44:28,900 were constrained in block length, you'd pick the best 751 00:44:28,900 --> 00:44:32,850 block code at the rate that you wanted, and the best 752 00:44:32,850 --> 00:44:35,980 minimum distance at that rate, and the best effective coding 753 00:44:35,980 --> 00:44:38,500 gain, and you'd do maximum likelihood decoding with 754 00:44:38,500 --> 00:44:40,560 trellis decoding. 755 00:44:40,560 --> 00:44:43,020 But if that's not usually the situation -- 756 00:44:43,020 --> 00:44:45,470 the situation in data communications, because 757 00:44:45,470 --> 00:44:50,230 usually more you have a stream of data or a packet length of 758 00:44:50,230 --> 00:44:54,605 at least 1,000, and then it would certainly be better to 759 00:44:54,605 --> 00:44:55,855 use convolutional codes. 760 00:44:55,855 --> 00:44:58,220 761 00:44:58,220 --> 00:44:59,850 And trellis decoding is -- 762 00:44:59,850 --> 00:45:01,860 as long as we're in this class of techniques. 763 00:45:01,860 --> 00:45:05,340 Where we're finally going in this course is to the new 764 00:45:05,340 --> 00:45:08,140 codes that have been developed in the past decade, 765 00:45:08,140 --> 00:45:12,090 capacity-approaching codes, turbo codes, low-density 766 00:45:12,090 --> 00:45:13,340 parity-check codes. 767 00:45:13,340 --> 00:45:15,640 768 00:45:15,640 --> 00:45:19,230 These are simple enough to do so that as soon as you get 769 00:45:19,230 --> 00:45:24,250 past the small codes, length less than 100, you would 770 00:45:24,250 --> 00:45:25,910 switch over to that track. 771 00:45:25,910 --> 00:45:28,872 You would use a capacity-approaching code. 772 00:45:28,872 --> 00:45:33,080 But we don't know about them yet. 773 00:45:33,080 --> 00:45:36,010 I guess that's the end of my lecture on this, so it's a 774 00:45:36,010 --> 00:45:40,130 very good place for questions about state-of-the-art, where 775 00:45:40,130 --> 00:45:41,400 we've been, where we're going. 776 00:45:41,400 --> 00:45:45,820 777 00:45:45,820 --> 00:45:47,560 The floor is always open. 778 00:45:47,560 --> 00:45:52,110 But that was all I was going to say about Chapter 10. 779 00:45:52,110 --> 00:45:58,340 780 00:45:58,340 --> 00:46:03,040 Chapter 11, Codes on Graphs. 781 00:46:03,040 --> 00:46:09,170 And now we begin our final stretch drive towards 782 00:46:09,170 --> 00:46:11,960 capacity-approaching codes, which is going to be the 783 00:46:11,960 --> 00:46:15,320 subject of the next three chapters. 784 00:46:15,320 --> 00:46:19,890 Chapter 11 is sort of the beginnings of codes on graphs. 785 00:46:19,890 --> 00:46:25,940 We're going to look at some elementary representations. 786 00:46:25,940 --> 00:46:27,960 One of the ones we're going to look at is a trellis 787 00:46:27,960 --> 00:46:28,840 representation. 788 00:46:28,840 --> 00:46:30,700 That's one of the reasons we've been going through this 789 00:46:30,700 --> 00:46:39,330 theory, is we're going to, again, encounter trellises as 790 00:46:39,330 --> 00:46:46,900 a natural, simple, cycle-free, graphical representation that, 791 00:46:46,900 --> 00:46:49,830 in some sense, is about as good as you can ever do, as 792 00:46:49,830 --> 00:46:51,815 long as you don't allow cycles in graphs. 793 00:46:51,815 --> 00:46:55,500 794 00:46:55,500 --> 00:46:59,110 But then we're going to go on to graphs with cycles, which 795 00:46:59,110 --> 00:47:01,320 is really the way to understand these 796 00:47:01,320 --> 00:47:03,580 capacity-approaching codes. 797 00:47:03,580 --> 00:47:06,370 So in this chapter, we'll start the subject, we'll 798 00:47:06,370 --> 00:47:09,000 introduce graphical representations of codes in 799 00:47:09,000 --> 00:47:12,141 more general terms. 800 00:47:12,141 --> 00:47:16,120 In Chapter 12, we'll talk about the generic decoding 801 00:47:16,120 --> 00:47:22,510 algorithms for codes on graphs called sum-product and min-sum 802 00:47:22,510 --> 00:47:25,720 decoding algorithms. 803 00:47:25,720 --> 00:47:31,830 And then in Chapter 13, we'll talk about the actual classes 804 00:47:31,830 --> 00:47:35,800 of capacity-approaching codes that people have developed, 805 00:47:35,800 --> 00:47:38,670 what their graphs are, how you decode them -- 806 00:47:38,670 --> 00:47:42,760 I hope in enough detail so that you'll get a good sense 807 00:47:42,760 --> 00:47:46,530 for how all this works. 808 00:47:46,530 --> 00:47:48,260 So that's where we're going. 809 00:47:48,260 --> 00:47:56,130 810 00:47:56,130 --> 00:47:59,100 What are we talking about when we're talking 811 00:47:59,100 --> 00:48:00,420 about codes on graphs? 812 00:48:00,420 --> 00:48:03,460 813 00:48:03,460 --> 00:48:08,760 There are many styles of representations of codes, and 814 00:48:08,760 --> 00:48:14,580 at this point, for linear codes, we have seen a number 815 00:48:14,580 --> 00:48:17,990 of ways we can characterize a code. 816 00:48:17,990 --> 00:48:21,130 One of the ways is just by giving a set of generators for 817 00:48:21,130 --> 00:48:23,880 the code, k generators for an n, k code. 818 00:48:23,880 --> 00:48:26,390 And that sort of characterizes the code. 819 00:48:26,390 --> 00:48:30,270 Another way is to basically give the generator matrix for 820 00:48:30,270 --> 00:48:34,880 the dual code, the h matrix, which becomes the parity-check 821 00:48:34,880 --> 00:48:37,030 matrix for the code. 822 00:48:37,030 --> 00:48:42,180 So we can also characterize the code by a 823 00:48:42,180 --> 00:48:43,540 dual-generator matrix. 824 00:48:43,540 --> 00:48:45,860 That's a particularly efficient way to do it if you 825 00:48:45,860 --> 00:48:50,190 have a high-rate code like a single parity-check code, it's 826 00:48:50,190 --> 00:48:55,450 better to say that the dual code is the repetition code 827 00:48:55,450 --> 00:49:00,160 and it's the set of all code words that are orthogonal to 828 00:49:00,160 --> 00:49:02,600 the all-one code word. 829 00:49:02,600 --> 00:49:05,745 That's the simplest characterization of the single 830 00:49:05,745 --> 00:49:06,380 parity-check code. 831 00:49:06,380 --> 00:49:10,962 In other words, that the sum of all bits is equal to 0. 832 00:49:10,962 --> 00:49:13,680 That's what orthogonal to the all-one code word means. 833 00:49:13,680 --> 00:49:16,920 So that's a simpler representation than giving a 834 00:49:16,920 --> 00:49:22,030 generator matrix formed for the single parity-check code. 835 00:49:22,030 --> 00:49:28,190 Or now we have this trellis representation, which 836 00:49:28,190 --> 00:49:30,870 certainly looks like a more graphical representation. 837 00:49:30,870 --> 00:49:35,150 It certainly characterizes the code in a very direct way. 838 00:49:35,150 --> 00:49:36,120 What does a trellis do? 839 00:49:36,120 --> 00:49:40,260 It basically displays all the code words on a graph. 840 00:49:40,260 --> 00:49:43,733 There's a 1:1 correspondence between the set of all paths, 841 00:49:43,733 --> 00:49:48,840 from the root to the n-node in the graph and the code. 842 00:49:48,840 --> 00:49:51,020 So that's another way to characterize the code. 843 00:49:51,020 --> 00:49:53,650 844 00:49:53,650 --> 00:49:56,960 So now we're going to go to higher level graphical 845 00:49:56,960 --> 00:49:58,210 representations. 846 00:49:58,210 --> 00:50:00,330 847 00:50:00,330 --> 00:50:09,910 As a first step, we talk about behavioral realizations. 848 00:50:09,910 --> 00:50:15,790 Again, a term from system theory, closely identified 849 00:50:15,790 --> 00:50:24,310 with Jan Willems, who promoted this way of representing 850 00:50:24,310 --> 00:50:26,400 linear systems. 851 00:50:26,400 --> 00:50:34,980 In linear system theory terms, the basic idea in behavioral 852 00:50:34,980 --> 00:50:40,030 systems theory is you describe a system by the set of all its 853 00:50:40,030 --> 00:50:41,530 possible trajectories. 854 00:50:41,530 --> 00:50:45,230 What are all the things it can do? 855 00:50:45,230 --> 00:50:48,180 And now, how do you characterize the trajectories? 856 00:50:48,180 --> 00:50:54,212 Often, you characterize them by a set of local constraints. 857 00:50:54,212 --> 00:50:59,170 You can think of it as equations that the system has 858 00:50:59,170 --> 00:51:01,900 to satisfy. 859 00:51:01,900 --> 00:51:05,110 Ordinarily, they're differential equations or 860 00:51:05,110 --> 00:51:06,960 partial differential equations or something. 861 00:51:06,960 --> 00:51:12,190 And if it satisfies all of those constraints, then it's a 862 00:51:12,190 --> 00:51:17,060 trajectory that satisfies all these partial differential 863 00:51:17,060 --> 00:51:18,830 equations, is a valid trajectory. 864 00:51:18,830 --> 00:51:23,356 865 00:51:23,356 --> 00:51:25,530 So you can set up the whole system that way. 866 00:51:25,530 --> 00:51:35,525 So it characterizes a system by its trajectories. 867 00:51:35,525 --> 00:51:38,170 868 00:51:38,170 --> 00:51:41,300 That's the fundamental thing. 869 00:51:41,300 --> 00:51:43,980 And it characterizes trajectories -- 870 00:51:43,980 --> 00:51:45,520 legitimate, valid trajectories -- 871 00:51:45,520 --> 00:51:49,610 872 00:51:49,610 --> 00:51:50,860 by local constraints. 873 00:51:50,860 --> 00:51:57,920 874 00:51:57,920 --> 00:52:00,350 Let me get a little bit more concrete. 875 00:52:00,350 --> 00:52:07,510 In coding terms, by trajectories we 876 00:52:07,510 --> 00:52:10,280 just mean code words. 877 00:52:10,280 --> 00:52:13,670 So this is a very compatible notion with the way we've been 878 00:52:13,670 --> 00:52:14,600 talking in coding. 879 00:52:14,600 --> 00:52:15,490 What is a code? 880 00:52:15,490 --> 00:52:18,480 It's simply a set of code words. 881 00:52:18,480 --> 00:52:21,900 So if we simply say, what are all the possible code words? 882 00:52:21,900 --> 00:52:23,870 We have to find all the trajectories of the code, and 883 00:52:23,870 --> 00:52:27,700 we have a very explicit example of that in trellises. 884 00:52:27,700 --> 00:52:29,120 We've called the paths 885 00:52:29,120 --> 00:52:32,750 trajectories through a trellis. 886 00:52:32,750 --> 00:52:33,680 What is a code? 887 00:52:33,680 --> 00:52:38,570 It's simply the set of all valid code words. 888 00:52:38,570 --> 00:52:43,640 And how do we characterize the valid code words? 889 00:52:43,640 --> 00:52:46,630 Well, in each of these styles -- 890 00:52:46,630 --> 00:52:50,850 the generator, parity-check, trellis style -- 891 00:52:50,850 --> 00:52:53,820 the code words are the code words that satisfy certain 892 00:52:53,820 --> 00:52:56,940 equations or constraints. 893 00:52:56,940 --> 00:52:59,756 For instance, the generator representation -- 894 00:52:59,756 --> 00:53:04,050 895 00:53:04,050 --> 00:53:12,515 we've said a code is the set of all uG, such that u is in a 896 00:53:12,515 --> 00:53:14,930 field to the k. 897 00:53:14,930 --> 00:53:22,930 That's a generator matrix style of representing a code. 898 00:53:22,930 --> 00:53:29,580 In other words, we can say it's the set of all y, such 899 00:53:29,580 --> 00:53:41,210 that y equals uG for some u in Fk. 900 00:53:41,210 --> 00:53:45,680 901 00:53:45,680 --> 00:53:55,480 Or we can say it's all y, such that y minus uG equals 0 for 902 00:53:55,480 --> 00:54:01,040 some u in Fk. 903 00:54:01,040 --> 00:54:08,480 So it's the set of all y that, together with some auxiliary 904 00:54:08,480 --> 00:54:11,130 variables which are not part of the code word -- 905 00:54:11,130 --> 00:54:14,450 here they're input variables, if you like, or information 906 00:54:14,450 --> 00:54:17,470 bits, information symbols. 907 00:54:17,470 --> 00:54:20,090 We'll come to think of these as state variables because 908 00:54:20,090 --> 00:54:22,720 they're hidden, they're not visible, they're just an 909 00:54:22,720 --> 00:54:24,125 auxiliary part of the description. 910 00:54:24,125 --> 00:54:28,050 911 00:54:28,050 --> 00:54:30,570 They're code words that, together with some auxiliary 912 00:54:30,570 --> 00:54:37,710 variables that, in this case, can be freely chosen, satisfy 913 00:54:37,710 --> 00:54:40,800 a certain set of linear homogeneous equations. 914 00:54:40,800 --> 00:54:45,810 915 00:54:45,810 --> 00:54:49,935 Something that's even better in the behavioral style is a 916 00:54:49,935 --> 00:54:51,822 parity-check representation. 917 00:54:51,822 --> 00:54:53,244 Let me put that up. 918 00:54:53,244 --> 00:54:56,570 919 00:54:56,570 --> 00:54:58,820 In a parity-check representation, we say the 920 00:54:58,820 --> 00:55:08,980 code is the set of all y, such that y H transpose equals 0, 921 00:55:08,980 --> 00:55:12,495 where H is the generator matrix of the dual code. 922 00:55:12,495 --> 00:55:15,190 923 00:55:15,190 --> 00:55:18,380 This is sometimes called a kernel representation. 924 00:55:18,380 --> 00:55:20,695 y is in the kernel of this linear transformation. 925 00:55:20,695 --> 00:55:24,280 926 00:55:24,280 --> 00:55:28,240 We see H transpose is a linear transformation from n-tuples 927 00:55:28,240 --> 00:55:33,670 to n minus k-tuples, and the ones whose image is 0 under 928 00:55:33,670 --> 00:55:38,520 this transformation, the ones that map to 0 are legitimate 929 00:55:38,520 --> 00:55:39,590 code words. 930 00:55:39,590 --> 00:55:42,110 So this is a kernel representation, this is an 931 00:55:42,110 --> 00:55:44,810 image representation. 932 00:55:44,810 --> 00:55:47,170 Here, we view the code as the image of the linear 933 00:55:47,170 --> 00:55:53,160 transformation, which is characterized by g, where the 934 00:55:53,160 --> 00:55:55,500 inputs are free. 935 00:55:55,500 --> 00:55:58,520 But the parity-check representation is much more 936 00:55:58,520 --> 00:56:01,180 explicitly, we don't need any auxiliary variables here. 937 00:56:01,180 --> 00:56:04,580 It's simply the set of n-tuples that satisfy a 938 00:56:04,580 --> 00:56:06,655 certain set of linear homogeneous equations. 939 00:56:06,655 --> 00:56:10,110 940 00:56:10,110 --> 00:56:14,890 So this maybe is a place where we start and, of course, it's 941 00:56:14,890 --> 00:56:16,840 the place where Bob Gallager started when he did 942 00:56:16,840 --> 00:56:20,050 low-density parity-check codes, which are a principal 943 00:56:20,050 --> 00:56:22,800 class of capacity-approaching codes. 944 00:56:22,800 --> 00:56:26,300 945 00:56:26,300 --> 00:56:30,020 Then in either case, we characterize the code by, in 946 00:56:30,020 --> 00:56:33,410 one case, directly satisfying some equations. 947 00:56:33,410 --> 00:56:38,340 In the other case, here we say that the total behavior is the 948 00:56:38,340 --> 00:56:45,550 set of all y and u, such that y minus uG equals 0. 949 00:56:45,550 --> 00:56:49,350 950 00:56:49,350 --> 00:56:54,640 So we're going to take the set of all n-tuples and k-tuples, 951 00:56:54,640 --> 00:56:56,870 such that y minus uG equals 0. 952 00:56:56,870 --> 00:57:02,250 And we say that's the total behavior of the code. 953 00:57:02,250 --> 00:57:07,910 The set of all y's and u's that satisfy this equation is 954 00:57:07,910 --> 00:57:08,840 the total behavior. 955 00:57:08,840 --> 00:57:12,160 And then we'll say the code word is just the projection of 956 00:57:12,160 --> 00:57:13,450 the behavior on the y's. 957 00:57:13,450 --> 00:57:16,450 958 00:57:16,450 --> 00:57:18,870 In other words, it's the set of all y's, such that this is 959 00:57:18,870 --> 00:57:22,410 satisfied first some u. 960 00:57:22,410 --> 00:57:23,660 And that's an equivalent description. 961 00:57:23,660 --> 00:57:27,420 962 00:57:27,420 --> 00:57:38,800 So the general setup, going back and forth, is that we 963 00:57:38,800 --> 00:57:42,945 have the observed symbols. 964 00:57:42,945 --> 00:57:48,520 965 00:57:48,520 --> 00:57:51,970 And in the coding case, this will always just be the n 966 00:57:51,970 --> 00:57:54,420 symbols in the code word. 967 00:57:54,420 --> 00:57:59,720 968 00:57:59,720 --> 00:58:03,340 These are the ones that we really care about. 969 00:58:03,340 --> 00:58:07,140 But we're also going to have some hidden or auxiliary 970 00:58:07,140 --> 00:58:16,000 symbols or state variables, hidden variables, which, in 971 00:58:16,000 --> 00:58:19,200 this case, for instance, are the u's. 972 00:58:19,200 --> 00:58:27,160 And we can introduce these at our convenience, just to make 973 00:58:27,160 --> 00:58:28,410 a good realization. 974 00:58:28,410 --> 00:58:31,930 975 00:58:31,930 --> 00:58:36,820 So we have two types of symbols: the observed, or 976 00:58:36,820 --> 00:58:38,400 external variables-- 977 00:58:38,400 --> 00:58:39,680 sometimes these are called external 978 00:58:39,680 --> 00:58:42,730 variables, internal variables. 979 00:58:42,730 --> 00:58:44,430 Again, in system theory, you're 980 00:58:44,430 --> 00:58:46,760 consistently seeing this. 981 00:58:46,760 --> 00:58:48,910 There are some that we can actually 982 00:58:48,910 --> 00:58:50,720 observe, interact with. 983 00:58:50,720 --> 00:58:53,540 These are the observed symbols, and then there are 984 00:58:53,540 --> 00:58:56,550 internal variables to the system, very often called 985 00:58:56,550 --> 00:59:01,960 state variables, which we can't directly see but which 986 00:59:01,960 --> 00:59:06,340 are part of the description of the system, as in this case. 987 00:59:06,340 --> 00:59:11,100 The whole system, we find the set of all internal and 988 00:59:11,100 --> 00:59:13,735 external variables that satisfy a set of constraints. 989 00:59:13,735 --> 00:59:19,140 990 00:59:19,140 --> 00:59:24,935 But we only care about the patterns of observed variables 991 00:59:24,935 --> 00:59:31,040 that are consistent with some set of internal variables. 992 00:59:31,040 --> 00:59:35,440 So the final element here in this general setup is 993 00:59:35,440 --> 00:59:44,370 constraints, which in this case are simply linear 994 00:59:44,370 --> 00:59:55,420 homogeneous equations on subsets of variables. 995 00:59:55,420 --> 01:00:07,250 996 01:00:07,250 --> 01:00:11,760 So let me go through similar kinds of examples, 997 01:00:11,760 --> 01:00:16,480 as I do in the notes. 998 01:00:16,480 --> 01:00:20,430 But let me maybe build up from the simplest one first, 999 01:00:20,430 --> 01:00:27,760 parity-check representation of our favorite 8, 4, 4 code. 1000 01:00:27,760 --> 01:00:32,870 In this case, the parity-check matrix is the same, we can 1001 01:00:32,870 --> 01:00:36,650 take it to be the same as the generator matrix. 1002 01:00:36,650 --> 01:00:40,730 Since, by now, we like trellis-oriented generator 1003 01:00:40,730 --> 01:00:42,590 matrices, we'll take that one. 1004 01:00:42,590 --> 01:00:48,900 1005 01:00:48,900 --> 01:00:55,730 So to get very explicit, so the code is the set of all y, 1006 01:00:55,730 --> 01:01:01,750 such that y H transpose equals 0, and y is 8-tuple. 1007 01:01:01,750 --> 01:01:08,510 1008 01:01:08,510 --> 01:01:11,500 So it's the set of all code words that satisfy these 1009 01:01:11,500 --> 01:01:12,750 parity-check constraints. 1010 01:01:12,750 --> 01:01:20,015 1011 01:01:20,015 --> 01:01:26,840 The first one is y1 plus y2 plus y3 plus y4 equals 0. 1012 01:01:26,840 --> 01:01:31,670 That's orthogonality to this first code word. 1013 01:01:31,670 --> 01:01:40,760 The second one is y2 plus y4 plus y5 plus y7 equals 0. 1014 01:01:40,760 --> 01:01:48,940 The third equation is y3 plus y4 plus y5 plus y6 equals 0. 1015 01:01:48,940 --> 01:01:59,310 And the fourth is y5 plus y6 plus y7 plus y8 equals 0. 1016 01:01:59,310 --> 01:02:02,920 If I have any 8-tuple y, such that the elements of the 1017 01:02:02,920 --> 01:02:06,000 8-tuples satisfy these four equations, then it's a code 1018 01:02:06,000 --> 01:02:08,420 word, if and only if. 1019 01:02:08,420 --> 01:02:10,740 A very explicit description of the code. 1020 01:02:10,740 --> 01:02:14,550 1021 01:02:14,550 --> 01:02:22,895 Now let's go onto a graph of behavioral realization. 1022 01:02:22,895 --> 01:02:30,760 1023 01:02:30,760 --> 01:02:37,720 So it's very natural to draw the following kind of graph of 1024 01:02:37,720 --> 01:02:39,680 this realization. 1025 01:02:39,680 --> 01:02:40,930 We let the y's -- 1026 01:02:40,930 --> 01:02:44,120 1027 01:02:44,120 --> 01:02:48,160 y1, y2, y3, y4 -- 1028 01:02:48,160 --> 01:02:51,360 sorry, I've changed the index set, haven't I? 1029 01:02:51,360 --> 01:02:53,920 I get 1 through 8 now. 1030 01:02:53,920 --> 01:03:00,910 I'm just going to make these into vertices of my graph. 1031 01:03:00,910 --> 01:03:05,020 1032 01:03:05,020 --> 01:03:11,480 So here we have symbols, and I'm going to make a bipartite 1033 01:03:11,480 --> 01:03:16,190 graph, and over here I'm going to have equations, or 1034 01:03:16,190 --> 01:03:21,140 constraints, or checks. 1035 01:03:21,140 --> 01:03:27,340 And I have four checks, which I'm going to draw like this. 1036 01:03:27,340 --> 01:03:33,490 1037 01:03:33,490 --> 01:03:38,440 The first check is that y1 plus y2 plus y3 plus y4 equals 1038 01:03:38,440 --> 01:03:40,490 0, so I'll just draw that like this. 1039 01:03:40,490 --> 01:03:43,800 1040 01:03:43,800 --> 01:03:45,660 That means the same thing as this. 1041 01:03:45,660 --> 01:03:50,460 1042 01:03:50,460 --> 01:03:58,520 The second one is y2 plus y4 plus y5 plus y7 equals 0. 1043 01:03:58,520 --> 01:04:02,580 And the third one is these four -- 1044 01:04:02,580 --> 01:04:06,420 two, three, four. 1045 01:04:06,420 --> 01:04:08,883 And the last one is these four. 1046 01:04:08,883 --> 01:04:13,230 This, this, this, this. 1047 01:04:13,230 --> 01:04:15,960 It's done more nicely in the notes. 1048 01:04:15,960 --> 01:04:24,130 This is called a Tanner graph, after an extremely good paper 1049 01:04:24,130 --> 01:04:31,410 by Michael Tanner in 1981, which was about the only paper 1050 01:04:31,410 --> 01:04:34,300 of any consequence in this subject between Gallager's 1051 01:04:34,300 --> 01:04:38,120 thesis in 1961 and the rediscovery of low-density 1052 01:04:38,120 --> 01:04:43,370 parity-check codes around 1995. 1053 01:04:43,370 --> 01:04:46,100 That's a graphical picture of these equations. 1054 01:04:46,100 --> 01:04:48,280 And what do we think of? 1055 01:04:48,280 --> 01:04:51,250 You can think of testing an 8-tuple. 1056 01:04:51,250 --> 01:04:56,560 Take an arbitrary, binary 8-tuple, think of these as 1057 01:04:56,560 --> 01:05:00,310 memory elements, put the bits of that 8-tuple in here, and 1058 01:05:00,310 --> 01:05:04,370 then ask if all these checks are satisfied. 1059 01:05:04,370 --> 01:05:06,720 Since we're talking mod 2 arithmetic, we're basically 1060 01:05:06,720 --> 01:05:11,500 asking if there are an even number of 1s attached to each 1061 01:05:11,500 --> 01:05:14,340 of these checks. 1062 01:05:14,340 --> 01:05:16,940 If and only if that's true, we've got a code 8-tuple. 1063 01:05:16,940 --> 01:05:19,520 1064 01:05:19,520 --> 01:05:23,750 From the set of all 256 8-tuples, we find 16 that 1065 01:05:23,750 --> 01:05:28,030 actually satisfy these checks, and that's the code. 1066 01:05:28,030 --> 01:05:29,400 That's one way of describing the code. 1067 01:05:29,400 --> 01:05:32,440 1068 01:05:32,440 --> 01:05:35,150 So that's how the graph is associated with local 1069 01:05:35,150 --> 01:05:36,080 constraints. 1070 01:05:36,080 --> 01:05:39,700 Well call them local, each of these constraints, because 1071 01:05:39,700 --> 01:05:44,240 each of them only involves four of the symbols. 1072 01:05:44,240 --> 01:05:48,670 In a much larger graph, a low-density parity-check code, 1073 01:05:48,670 --> 01:05:51,586 the idea is that there are not very many symbols that are 1074 01:05:51,586 --> 01:05:53,200 checked by each check. 1075 01:05:53,200 --> 01:05:57,430 The graph is sparse, in that sense. 1076 01:05:57,430 --> 01:05:59,110 But we're not quite there yet. 1077 01:05:59,110 --> 01:06:03,640 1078 01:06:03,640 --> 01:06:07,080 So that's pretty simple. 1079 01:06:07,080 --> 01:06:11,105 Let me do a generator representation. 1080 01:06:11,105 --> 01:06:16,620 This is going to be a little bit more complicated because 1081 01:06:16,620 --> 01:06:19,070 we have these auxiliary input variables. 1082 01:06:19,070 --> 01:06:21,830 1083 01:06:21,830 --> 01:06:26,870 But that doesn't really complicate it very much. 1084 01:06:26,870 --> 01:06:31,660 I keep using this 8, 4, 4 code just to emphasize that a 1085 01:06:31,660 --> 01:06:34,600 single code can be described in many different ways. 1086 01:06:34,600 --> 01:06:37,260 We're going to wind up with half a dozen different 1087 01:06:37,260 --> 01:06:42,830 graphical representations of this code, and they can be 1088 01:06:42,830 --> 01:06:44,946 grouped according to style. 1089 01:06:44,946 --> 01:06:50,625 1090 01:06:50,625 --> 01:06:54,370 So again, we could use our favorite generator matrix for 1091 01:06:54,370 --> 01:06:56,336 our favorite code. 1092 01:06:56,336 --> 01:06:57,625 It looks like that. 1093 01:06:57,625 --> 01:07:00,660 1094 01:07:00,660 --> 01:07:02,010 Now what do we mean? 1095 01:07:02,010 --> 01:07:04,770 What are our constraints now? 1096 01:07:04,770 --> 01:07:12,380 Well, now we have some hidden variables, ui, and we have y1 1097 01:07:12,380 --> 01:07:14,010 is equal to u1. 1098 01:07:14,010 --> 01:07:18,820 We're going to basically multiply this by u1. 1099 01:07:18,820 --> 01:07:23,820 I've got some u1, u2, u3, u4 multiplying this. 1100 01:07:23,820 --> 01:07:36,600 So y1 is u1, y2 is u1 plus u2, y3 is equal to u1 plus u3, y4 1101 01:07:36,600 --> 01:07:42,430 is equal to u1 plus u2 plus u3, so forth. 1102 01:07:42,430 --> 01:07:51,930 y5 is u2 plus u3 plus u4, y6 is u2 plus u4. 1103 01:07:51,930 --> 01:07:55,070 1104 01:07:55,070 --> 01:07:56,710 I jumped one here. 1105 01:07:56,710 --> 01:08:03,070 This is u3 plus u4, y7 is u2 plus u4, and y8 equals u8. 1106 01:08:03,070 --> 01:08:06,730 1107 01:08:06,730 --> 01:08:10,350 And you will agree that the code is described, if I can 1108 01:08:10,350 --> 01:08:17,060 find any pair, (y,u), such that this is satisfied, then y 1109 01:08:17,060 --> 01:08:18,310 is an element of the code. 1110 01:08:18,310 --> 01:08:21,540 1111 01:08:21,540 --> 01:08:25,160 I can choose u freely, that is the input. 1112 01:08:25,160 --> 01:08:29,560 If I find a y at any u, such that these equations are 1113 01:08:29,560 --> 01:08:33,399 satisfied, then I've found a code word. 1114 01:08:33,399 --> 01:08:35,130 And that's an if and only if. 1115 01:08:35,130 --> 01:08:38,300 So that's another characterization of the code. 1116 01:08:38,300 --> 01:08:46,380 And its Tanner graph looks like this. 1117 01:08:46,380 --> 01:08:49,580 Here in this case, we have hidden variables. 1118 01:08:49,580 --> 01:08:53,060 So let's put the inputs-- 1119 01:08:53,060 --> 01:08:57,540 or, we'll see they can also be thought of as states, but 1120 01:08:57,540 --> 01:08:59,210 they're hidden variables that we don't actually 1121 01:08:59,210 --> 01:09:00,790 see in the code word. 1122 01:09:00,790 --> 01:09:05,689 u1, u2, u3, u4. 1123 01:09:05,689 --> 01:09:11,310 You can think of these as free-driving variables. 1124 01:09:11,310 --> 01:09:16,950 And to get y1, here's what we do. 1125 01:09:16,950 --> 01:09:20,920 We're going to take y1 and we're going to create a mod 2 1126 01:09:20,920 --> 01:09:23,243 sum of some of these inputs. 1127 01:09:23,243 --> 01:09:26,080 In this case, it's just u1, so I can draw it 1128 01:09:26,080 --> 01:09:27,720 as a straight through. 1129 01:09:27,720 --> 01:09:33,649 But in general, I'm going to have to make a combination to 1130 01:09:33,649 --> 01:09:35,020 get all of these. 1131 01:09:35,020 --> 01:09:38,100 So I have eight symbols over here -- 1132 01:09:38,100 --> 01:09:40,870 code symbols -- 1133 01:09:40,870 --> 01:09:46,170 and again, some constraints that have to be satisfied. 1134 01:09:46,170 --> 01:10:00,840 Equations, mod 2 sums, y8. 1135 01:10:00,840 --> 01:10:04,650 And I just, again, draw a picture. 1136 01:10:04,650 --> 01:10:10,380 y2 is u1 plus u2, so u1 plus u2. 1137 01:10:10,380 --> 01:10:13,260 these sum to give y2. 1138 01:10:13,260 --> 01:10:17,560 And so, again, if I put down some u's, put down some y's, 1139 01:10:17,560 --> 01:10:23,320 test whether all these sums are correct, then I will have 1140 01:10:23,320 --> 01:10:29,150 tested whether I found a valid behavior or trajectory. 1141 01:10:29,150 --> 01:10:33,010 Next one is u1 plus u3. 1142 01:10:33,010 --> 01:10:38,820 The next one is u1, u2, u3. 1143 01:10:38,820 --> 01:10:43,580 The next one is u2, u3, u4. 1144 01:10:43,580 --> 01:10:47,960 The next one is u3, u4. 1145 01:10:47,960 --> 01:10:51,440 The next one is u2, u4. 1146 01:10:51,440 --> 01:10:53,178 And the last one is just that one. 1147 01:10:53,178 --> 01:10:57,710 1148 01:10:57,710 --> 01:11:01,750 OK, so that's another graph whose constraints 1149 01:11:01,750 --> 01:11:03,150 describe the code word. 1150 01:11:03,150 --> 01:11:06,060 1151 01:11:06,060 --> 01:11:08,740 Here it's helpful to start to introduce the following 1152 01:11:08,740 --> 01:11:09,360 convention. 1153 01:11:09,360 --> 01:11:12,840 We fill in the circle of the variables that we actually 1154 01:11:12,840 --> 01:11:16,690 want to see, and we leave these open, the free 1155 01:11:16,690 --> 01:11:17,940 variables over here. 1156 01:11:17,940 --> 01:11:21,970 1157 01:11:21,970 --> 01:11:24,440 Some people would call it a generalized Tanner graph. 1158 01:11:24,440 --> 01:11:29,290 In the notes, I just call this Tanner graph 2. 1159 01:11:29,290 --> 01:11:32,290 Tanner didn't actually consider state variables. 1160 01:11:32,290 --> 01:11:36,570 1161 01:11:36,570 --> 01:11:40,376 This was introduced into graphical models by a guy 1162 01:11:40,376 --> 01:11:45,990 named Wiberg in his PhD thesis in Sweden in about 1995. 1163 01:11:45,990 --> 01:11:48,320 And that was a huge contribution, to have hidden 1164 01:11:48,320 --> 01:11:51,360 variables as well as observed variables, because otherwise 1165 01:11:51,360 --> 01:11:54,150 you couldn't really draw a generator representation as a 1166 01:11:54,150 --> 01:11:57,640 graph, or a trellis representation. 1167 01:11:57,640 --> 01:11:58,910 You need hidden state variables. 1168 01:11:58,910 --> 01:12:01,720 1169 01:12:01,720 --> 01:12:07,810 So those are two graphical styles, and the idea here is 1170 01:12:07,810 --> 01:12:13,860 that every set of 12 variables, such that these 1171 01:12:13,860 --> 01:12:17,090 eight equations are satisfied, gives you a 1172 01:12:17,090 --> 01:12:18,340 legitimate code word. 1173 01:12:18,340 --> 01:12:21,040 1174 01:12:21,040 --> 01:12:25,650 Here you can easily think of this as -- 1175 01:12:25,650 --> 01:12:28,650 you notice that we've described these as passive 1176 01:12:28,650 --> 01:12:29,900 constraints. 1177 01:12:29,900 --> 01:12:31,720 1178 01:12:31,720 --> 01:12:35,020 It's not an input-output, block diagram type of 1179 01:12:35,020 --> 01:12:36,680 representation. 1180 01:12:36,680 --> 01:12:40,490 It causes cause-effects. 1181 01:12:40,490 --> 01:12:43,310 In this case, it implicitly is. 1182 01:12:43,310 --> 01:12:46,940 How would you actually implement this generator 1183 01:12:46,940 --> 01:12:49,400 representation? 1184 01:12:49,400 --> 01:12:52,980 You implement it by picking four input bits and then 1185 01:12:52,980 --> 01:12:55,230 seeing what code words they generate. 1186 01:12:55,230 --> 01:12:58,910 In other words, there's a very definite input-output relation 1187 01:12:58,910 --> 01:13:03,220 here, where we could make this into a directed graph by 1188 01:13:03,220 --> 01:13:05,900 drawing arrows here. 1189 01:13:05,900 --> 01:13:09,030 In general, in our graphical models, we're 1190 01:13:09,030 --> 01:13:10,140 not going to do this. 1191 01:13:10,140 --> 01:13:15,490 The behavioral style is very much not in this spirit. 1192 01:13:15,490 --> 01:13:18,970 But in this case, we clearly can. 1193 01:13:18,970 --> 01:13:22,690 So if you actually wanted to generate all the code words -- 1194 01:13:22,690 --> 01:13:26,440 that's why it's called a generator representation -- 1195 01:13:26,440 --> 01:13:30,430 you simply jump through all 16 possibilities here and you 1196 01:13:30,430 --> 01:13:34,300 generate all 16 code words. 1197 01:13:34,300 --> 01:13:37,430 Or if you wanted to do a simulation, generate code 1198 01:13:37,430 --> 01:13:40,630 words, this is the way you would do it. 1199 01:13:40,630 --> 01:13:44,530 This has easy cause and effect style. 1200 01:13:44,530 --> 01:13:48,530 The kernel representation is much more implicit. 1201 01:13:48,530 --> 01:13:52,930 What is the cause and the effect here? 1202 01:13:52,930 --> 01:13:56,960 In this case, it's actually possible to find a cause and 1203 01:13:56,960 --> 01:14:01,380 effect representation. 1204 01:14:01,380 --> 01:14:07,190 It turns out that these four bits can be taken as 1205 01:14:07,190 --> 01:14:10,740 information bits, or these four places form an 1206 01:14:10,740 --> 01:14:13,960 information set. 1207 01:14:13,960 --> 01:14:19,620 So suppose we choose these four bits, and it turns out 1208 01:14:19,620 --> 01:14:25,570 that for this graph, we can trace through as follows. 1209 01:14:25,570 --> 01:14:29,660 These three bits going into this zero-sum, 1210 01:14:29,660 --> 01:14:32,600 these all go in. 1211 01:14:32,600 --> 01:14:34,380 That determines this output, right? 1212 01:14:34,380 --> 01:14:38,710 1213 01:14:38,710 --> 01:14:42,890 d minus 1, if d is the degree of the zero-sum node, any d 1214 01:14:42,890 --> 01:14:48,750 minus 1 inputs determine the last output, because the 1215 01:14:48,750 --> 01:14:50,200 parity-check has to check. 1216 01:14:50,200 --> 01:14:54,270 So that determines this one, which then determines y4. 1217 01:14:54,270 --> 01:14:59,070 1218 01:14:59,070 --> 01:15:04,594 Now, knowing that, we know all but one input here, because I 1219 01:15:04,594 --> 01:15:06,540 also have this one. 1220 01:15:06,540 --> 01:15:11,180 And that determines this, and I think it probably also 1221 01:15:11,180 --> 01:15:14,370 determines this. 1222 01:15:14,370 --> 01:15:18,170 I now have three of the four inputs here, 1223 01:15:18,170 --> 01:15:20,750 so I can get this. 1224 01:15:20,750 --> 01:15:26,970 And anyway, by tracing through it I can create a directed 1225 01:15:26,970 --> 01:15:29,270 graph that gives me a cause and effect 1226 01:15:29,270 --> 01:15:31,920 relationship from here. 1227 01:15:31,920 --> 01:15:36,110 But it's much more implicit and it can't always be done. 1228 01:15:36,110 --> 01:15:40,460 Sometimes you can find an information set that works 1229 01:15:40,460 --> 01:15:42,950 with these particular set of equations and sometimes you 1230 01:15:42,950 --> 01:15:46,440 can't, sometimes you get stuck. 1231 01:15:46,440 --> 01:15:49,050 We can come back to this when we talk about binary erasure 1232 01:15:49,050 --> 01:15:50,300 channel and stopping sets. 1233 01:15:50,300 --> 01:15:53,270 1234 01:15:53,270 --> 01:15:56,180 In this case, given these four, you can determine the 1235 01:15:56,180 --> 01:15:58,845 remaining four. 1236 01:15:58,845 --> 01:16:05,420 If you think of it in equation terms, if you're given y1, y2, 1237 01:16:05,420 --> 01:16:08,530 y3, you can basically go through a Gaussian elimination 1238 01:16:08,530 --> 01:16:14,325 of these equations and find y4, y6, y7 and y8. 1239 01:16:14,325 --> 01:16:17,080 1240 01:16:17,080 --> 01:16:21,220 So you can regard y1, y2, y3, and y5 as inputs. 1241 01:16:21,220 --> 01:16:24,170 1242 01:16:24,170 --> 01:16:27,270 This is much more the behavioral style, where it's 1243 01:16:27,270 --> 01:16:32,800 just simply any n-tuple that satisfies the constraints is a 1244 01:16:32,800 --> 01:16:34,780 legitimate n-tuple. 1245 01:16:34,780 --> 01:16:38,100 This is more cause and effect style over here. 1246 01:16:38,100 --> 01:16:42,360 1247 01:16:42,360 --> 01:16:46,610 Since we're at the end, let me just introduce one more thing. 1248 01:16:46,610 --> 01:16:51,450 1249 01:16:51,450 --> 01:16:53,170 This is a bipartite graph. 1250 01:16:53,170 --> 01:16:56,980 We basically have a graph with two types of nodes, one 1251 01:16:56,980 --> 01:17:01,640 representing variables, one representing constraints. 1252 01:17:01,640 --> 01:17:06,980 And the edges always go between a variable and a 1253 01:17:06,980 --> 01:17:08,520 constraint. 1254 01:17:08,520 --> 01:17:12,010 So that's what we mean by a bipartite graph. 1255 01:17:12,010 --> 01:17:14,690 There are two types of vertices, and all the edges 1256 01:17:14,690 --> 01:17:20,240 connect one type of vertex to another type of vertex. 1257 01:17:20,240 --> 01:17:22,420 So is this over here. 1258 01:17:22,420 --> 01:17:26,932 I've drawn it in a way where it's not so obvious. 1259 01:17:26,932 --> 01:17:30,040 In a generalized Tanner graph, you'd put all the variables 1260 01:17:30,040 --> 01:17:31,900 over on one side and the constraints over 1261 01:17:31,900 --> 01:17:32,600 on the other side. 1262 01:17:32,600 --> 01:17:38,800 In this case, these observed variables go naturally with 1263 01:17:38,800 --> 01:17:40,950 the constraints, so I put them over here. 1264 01:17:40,950 --> 01:17:43,730 But I could have put them over here and you would see this is 1265 01:17:43,730 --> 01:17:45,600 also a bipartite graph. 1266 01:17:45,600 --> 01:17:48,560 1267 01:17:48,560 --> 01:17:52,870 So this is the general character. 1268 01:17:52,870 --> 01:17:55,970 What do we think of the edges as doing in this graph? 1269 01:17:55,970 --> 01:18:00,265 We think of the edges as conducting the variables that 1270 01:18:00,265 --> 01:18:02,510 they're associated with. 1271 01:18:02,510 --> 01:18:09,050 So all of these edges, in fact, convey u1 to this 1272 01:18:09,050 --> 01:18:10,490 constraint. 1273 01:18:10,490 --> 01:18:14,030 We could, if we wanted to, label all of the edges with 1274 01:18:14,030 --> 01:18:15,890 their associated variables. 1275 01:18:15,890 --> 01:18:18,530 That's what they mean. 1276 01:18:18,530 --> 01:18:24,270 They mean here that u1 equals y1, or that u1 1277 01:18:24,270 --> 01:18:27,880 plus u2 equals y2. 1278 01:18:27,880 --> 01:18:33,580 So the edges just serve for communication in this graph. 1279 01:18:33,580 --> 01:18:38,000 There's another style of graph, which I introduced, and 1280 01:18:38,000 --> 01:18:40,450 therefore, people -- 1281 01:18:40,450 --> 01:18:43,470 not me -- but other people have called it a Forney graph. 1282 01:18:43,470 --> 01:18:44,725 I call it a normal graph. 1283 01:18:44,725 --> 01:18:48,940 1284 01:18:48,940 --> 01:18:56,520 The difference is that basically the vertices all 1285 01:18:56,520 --> 01:19:05,365 equal constraints, and the edges represent variables. 1286 01:19:05,365 --> 01:19:09,490 1287 01:19:09,490 --> 01:19:14,050 And it has certain advantages that have led to it being used 1288 01:19:14,050 --> 01:19:14,650 more and more. 1289 01:19:14,650 --> 01:19:20,770 Let me just indicate how we go from a graph of this style to 1290 01:19:20,770 --> 01:19:22,610 a graph of this style. 1291 01:19:22,610 --> 01:19:26,550 We basically just make all the variables into equality 1292 01:19:26,550 --> 01:19:28,590 constraints. 1293 01:19:28,590 --> 01:19:40,210 So in this case, I draw y1, y2, and so forth. 1294 01:19:40,210 --> 01:19:43,580 I put a little dongle on them to indicate that they 1295 01:19:43,580 --> 01:19:46,450 communicate with the outside world. 1296 01:19:46,450 --> 01:19:49,000 They are observed, external variables. 1297 01:19:49,000 --> 01:19:52,110 1298 01:19:52,110 --> 01:19:54,840 And then I put a little equality constraint that says 1299 01:19:54,840 --> 01:20:00,100 all of the variables attached to this constraint have to be 1300 01:20:00,100 --> 01:20:03,430 the same, which is really doing the same thing. 1301 01:20:03,430 --> 01:20:07,630 Otherwise, the topology of the graph looks the same. 1302 01:20:07,630 --> 01:20:12,560 These are, again, zero-sum constraints, or parity-check 1303 01:20:12,560 --> 01:20:15,400 constraints, and the connections are 1304 01:20:15,400 --> 01:20:17,520 done just as before. 1305 01:20:17,520 --> 01:20:33,475 1306 01:20:33,475 --> 01:20:35,700 I forget. 1307 01:20:35,700 --> 01:20:37,750 I'm not going to do them all out. 1308 01:20:37,750 --> 01:20:42,100 The only change, in this case, is to replace external 1309 01:20:42,100 --> 01:20:44,710 variables by an equality constraint in this. 1310 01:20:44,710 --> 01:20:47,930 So the equality constraint basically says this edge still 1311 01:20:47,930 --> 01:20:52,440 equals y1, this edge still equals y2. 1312 01:20:52,440 --> 01:20:54,930 That's what the equality constraint means. 1313 01:20:54,930 --> 01:20:56,470 Everything tied to that equality 1314 01:20:56,470 --> 01:20:58,630 constraint has to be equal. 1315 01:20:58,630 --> 01:21:03,260 So we can think of these as just replicas of this original 1316 01:21:03,260 --> 01:21:06,280 variable out here. 1317 01:21:06,280 --> 01:21:10,960 So far, you don't see any reason to prefer this. 1318 01:21:10,960 --> 01:21:18,100 If I do it in this graph, then, again, I get equality 1319 01:21:18,100 --> 01:21:20,090 constraints here. 1320 01:21:20,090 --> 01:21:26,958 I get zero-sum constraints, again, over here. 1321 01:21:26,958 --> 01:21:28,425 Eight of them. 1322 01:21:28,425 --> 01:21:31,360 1323 01:21:31,360 --> 01:21:34,070 And in this case, I can just represent the external 1324 01:21:34,070 --> 01:21:38,390 variables directly by these little dongles. 1325 01:21:38,390 --> 01:21:40,487 And again, I have the same graph here. 1326 01:21:40,487 --> 01:21:43,680 1327 01:21:43,680 --> 01:21:45,950 You still probably don't see that there's very much 1328 01:21:45,950 --> 01:21:47,420 difference and, in fact, I'd agree. 1329 01:21:47,420 --> 01:21:52,620 There's hardly any difference between these two styles. 1330 01:21:52,620 --> 01:21:53,870 So forth. 1331 01:21:53,870 --> 01:22:00,230 1332 01:22:00,230 --> 01:22:03,030 This is the normal graph that's equivalent to this 1333 01:22:03,030 --> 01:22:06,140 Tanner graph, it's obtained in the same way. 1334 01:22:06,140 --> 01:22:09,020 If I wanted to do it more painstakingly, I would put 1335 01:22:09,020 --> 01:22:12,030 equality constraints for each of these, I'd put little 1336 01:22:12,030 --> 01:22:14,990 external variables, and then I'd see I don't really need 1337 01:22:14,990 --> 01:22:16,260 the equality constraints. 1338 01:22:16,260 --> 01:22:20,810 So I can just compress it into y1 through y8. 1339 01:22:20,810 --> 01:22:25,590 1340 01:22:25,590 --> 01:22:29,680 The advantages, I will say, of the normal graph will appear 1341 01:22:29,680 --> 01:22:30,690 as we go along. 1342 01:22:30,690 --> 01:22:36,280 The main advantage is that you can prove a very nice duality 1343 01:22:36,280 --> 01:22:39,340 theorem for normal graphs -- 1344 01:22:39,340 --> 01:22:44,830 that basically shows the duality between generator and 1345 01:22:44,830 --> 01:22:46,080 parity-check constraints. 1346 01:22:46,080 --> 01:22:48,750 But to say anything about that right now would just 1347 01:22:48,750 --> 01:22:54,200 hand-waving, so we'll come back to this next week. 1348 01:22:54,200 --> 01:22:56,710 See you in a week, have a nice weekend. 1349 01:22:56,710 --> 01:23:06,282