1 00:00:00,000 --> 00:00:01,930 PROFESSOR: --wind up doing a substantial part of Chapter 2 00:00:01,930 --> 00:00:04,000 11, although I do tend to be over-optimistic. 3 00:00:06,810 --> 00:00:11,830 So we're handing out Chapter 11, which I expect to revise 4 00:00:11,830 --> 00:00:13,810 still further. 5 00:00:13,810 --> 00:00:17,110 As we get towards these end chapters, these are things 6 00:00:17,110 --> 00:00:21,820 I've only taught from, at most, once before, and so I 7 00:00:21,820 --> 00:00:24,210 have more revisions to do than the earlier chapters which 8 00:00:24,210 --> 00:00:27,580 I've used for many years. 9 00:00:27,580 --> 00:00:31,660 The Problem Set Seven, as we announced last time, you only 10 00:00:31,660 --> 00:00:34,010 needed to hand in the first four problems today, so we're 11 00:00:34,010 --> 00:00:37,330 only handing out the solutions for the first four today. 12 00:00:37,330 --> 00:00:40,090 I haven't made up any additional problems, so 13 00:00:40,090 --> 00:00:43,350 Problem Set Eight is simply to do the last two problems in 14 00:00:43,350 --> 00:00:46,490 Problem Set Seven, due next Wednesday. 15 00:00:46,490 --> 00:00:50,130 And a reminder that we have no class on Monday because of 16 00:00:50,130 --> 00:00:53,870 this peculiar local holiday we have called Patriot's Day, 17 00:00:53,870 --> 00:00:54,830 when we run the marathon. 18 00:00:54,830 --> 00:00:57,070 And I hope you'll all be out there running the marathon. 19 00:01:00,400 --> 00:01:02,095 So enjoy the long weekend. 20 00:01:05,330 --> 00:01:08,330 So what have we been doing in Chapter 10? 21 00:01:08,330 --> 00:01:18,230 We've basically been getting minimal trellis realizations, 22 00:01:18,230 --> 00:01:22,880 or state-state space realizations, in system theory 23 00:01:22,880 --> 00:01:29,960 terms, for linear block codes. 24 00:01:29,960 --> 00:01:36,670 We've been focusing on binary linear block codes, but 25 00:01:36,670 --> 00:01:43,950 clearly everything we've been doing goes to block codes over 26 00:01:43,950 --> 00:01:46,160 any finite field or, in fact, over the 27 00:01:46,160 --> 00:01:47,795 real or complex field. 28 00:01:47,795 --> 00:01:49,960 It's just basic realization theory, 29 00:01:49,960 --> 00:01:53,790 minimal realization theory. 30 00:01:53,790 --> 00:02:00,060 So as long as we have a linear time-varying system, it works. 31 00:02:00,060 --> 00:02:02,090 I had a question after class last time, well, doesn't it 32 00:02:02,090 --> 00:02:04,050 work just as well for nonlinear systems? 33 00:02:04,050 --> 00:02:07,720 Can't we do the trellis in the same way? 34 00:02:07,720 --> 00:02:08,990 In general, not. 35 00:02:08,990 --> 00:02:10,919 If it's a nonlinear system with a group 36 00:02:10,919 --> 00:02:12,585 property, then you can't. 37 00:02:12,585 --> 00:02:14,660 It's really the group property that's key to these 38 00:02:14,660 --> 00:02:17,260 constructions. 39 00:02:17,260 --> 00:02:20,320 And we seem to have pretty well cracked the problem. 40 00:02:20,320 --> 00:02:23,190 We have this tool called the trellis-oriented or 41 00:02:23,190 --> 00:02:26,830 minimal-span generator matrix, from which we can read all the 42 00:02:26,830 --> 00:02:28,540 parameters of the trellis, from which we 43 00:02:28,540 --> 00:02:30,430 can construct a trellis. 44 00:02:30,430 --> 00:02:34,240 Really, we found that the trellis-oriented generator 45 00:02:34,240 --> 00:02:37,950 matrix contains all the information we need to 46 00:02:37,950 --> 00:02:41,360 construct a minimal trellis. 47 00:02:41,360 --> 00:02:42,960 And it's just a matter of turning the crank 48 00:02:42,960 --> 00:02:44,210 once we have that. 49 00:02:46,620 --> 00:02:52,230 The only degree of freedom that I've allowed so far in 50 00:02:52,230 --> 00:02:56,240 trellis constructions is sectionalization. 51 00:02:56,240 --> 00:03:01,200 I've shown you that if we can choose where to put the state 52 00:03:01,200 --> 00:03:02,450 spaces wherever we like-- 53 00:03:05,210 --> 00:03:07,990 typically we put them in the center and other places, 54 00:03:07,990 --> 00:03:10,040 sometimes we put them everywhere we could put them, 55 00:03:10,040 --> 00:03:13,400 that's called an unsectionalized trellis-- 56 00:03:13,400 --> 00:03:19,570 and by so doing, we can sometimes reduce the apparent 57 00:03:19,570 --> 00:03:21,520 state-space complexity. 58 00:03:21,520 --> 00:03:25,340 But we can never reduce the branch complexity. 59 00:03:25,340 --> 00:03:30,030 So we're left with branch complexity as a fundamental 60 00:03:30,030 --> 00:03:32,520 complexity parameter for linear block 61 00:03:32,520 --> 00:03:35,890 codes, or so it seems. 62 00:03:35,890 --> 00:03:37,320 Is there any other-- 63 00:03:37,320 --> 00:03:38,141 yes? 64 00:03:38,141 --> 00:03:42,470 AUDIENCE: Does the complexity of the number of computations 65 00:03:42,470 --> 00:03:45,452 you need to go through the trellis depends only on the 66 00:03:45,452 --> 00:03:46,856 number of branches, right? 67 00:03:46,856 --> 00:03:48,640 PROFESSOR: It depends primarily 68 00:03:48,640 --> 00:03:50,580 on the number branches. 69 00:03:50,580 --> 00:03:52,050 It also depends on the edges. 70 00:03:52,050 --> 00:03:59,140 If you go through a detailed operation count, I think I 71 00:03:59,140 --> 00:04:03,540 give the formula for if you count additions and 72 00:04:03,540 --> 00:04:08,130 comparisons as an addition and selection you don't count as a 73 00:04:08,130 --> 00:04:12,580 logical operation, then I think the total count for the 74 00:04:12,580 --> 00:04:17,630 Viterbi algorithm is twice the number of edges in the graph, 75 00:04:17,630 --> 00:04:20,529 which is the number of branches, minus the number of 76 00:04:20,529 --> 00:04:24,944 vertices, which you lose one for every-- 77 00:04:24,944 --> 00:04:26,375 AUDIENCE: What do you mean by vertices? 78 00:04:26,375 --> 00:04:28,960 PROFESSOR: By vertices I mean states, now I'm talking about 79 00:04:28,960 --> 00:04:30,430 it as a graph. 80 00:04:30,430 --> 00:04:32,630 Minus one or plus one or something. 81 00:04:32,630 --> 00:04:35,220 So that's an exact count for the Viterbi algorithm. 82 00:04:35,220 --> 00:04:39,560 But it's dominated by the total number of edges in the 83 00:04:39,560 --> 00:04:42,990 graph which, in turn, is well-estimated by the maximum 84 00:04:42,990 --> 00:04:45,020 branch complexity at any one time. 85 00:04:47,570 --> 00:04:51,640 So there's a bit of a trade-off between an exact 86 00:04:51,640 --> 00:04:57,676 count and just getting a gross handle on the situation. 87 00:04:57,676 --> 00:05:01,760 But I say it's certainly got to exceed the branch 88 00:05:01,760 --> 00:05:02,400 complexity. 89 00:05:02,400 --> 00:05:05,170 So when I find the branch complexity goes up 90 00:05:05,170 --> 00:05:08,650 exponentially, even for one branch, then I know that the 91 00:05:08,650 --> 00:05:10,950 total Viterbi algorithm complexity has to go up 92 00:05:10,950 --> 00:05:14,462 exponentially with that. 93 00:05:14,462 --> 00:05:16,805 All right? 94 00:05:16,805 --> 00:05:20,160 So is that the whole story? 95 00:05:20,160 --> 00:05:23,650 Do we now have a pretty definitive handle on the, 96 00:05:23,650 --> 00:05:28,200 quote, complexity of at least maximum likelihood decoding, 97 00:05:28,200 --> 00:05:31,800 trellis decoding of a block code. 98 00:05:31,800 --> 00:05:35,790 And if you've read Chapter 11, you know there's one other 99 00:05:35,790 --> 00:05:38,940 degree of freedom that we haven't exploited yet in 100 00:05:38,940 --> 00:05:41,730 representing a linear block code. 101 00:05:41,730 --> 00:05:44,570 Anyone read that far, or are you just going to wait for me 102 00:05:44,570 --> 00:05:46,720 to explain it to you? 103 00:05:46,720 --> 00:05:49,010 Flip the coordinates, very good. 104 00:05:49,010 --> 00:05:51,440 Could that make a difference? 105 00:05:51,440 --> 00:05:53,050 Yes, it certainly could. 106 00:05:53,050 --> 00:06:04,590 Let's take our standard example by now, the 8-4-4 code 107 00:06:04,590 --> 00:06:07,090 with trellis-oriented matrix that looks like this. 108 00:06:10,030 --> 00:06:14,370 OK, and we know everything about this by now. 109 00:06:14,370 --> 00:06:18,510 But suppose, say, we flip these two center coordinates. 110 00:06:18,510 --> 00:06:25,070 Certainly it would be an equivalent code from all of 111 00:06:25,070 --> 00:06:27,560 its n, k, d and that sort of properties. 112 00:06:27,560 --> 00:06:32,510 So I want to take these two coordinates and make them 1, 113 00:06:32,510 --> 00:06:36,290 1, 1, 0, 0, 1, 1, 1. 114 00:06:40,670 --> 00:06:43,210 Let's suppose that's a trellis-oriented generator 115 00:06:43,210 --> 00:06:44,260 matrix for the code. 116 00:06:44,260 --> 00:06:47,120 I think it is. 117 00:06:47,120 --> 00:06:52,940 It's got starting times here, here, here, and now here. 118 00:06:52,940 --> 00:06:59,570 And it's got ending times now here, here, here, and here. 119 00:06:59,570 --> 00:07:01,230 OK. 120 00:07:01,230 --> 00:07:06,020 Where I introduce the diverging and merging symbols 121 00:07:06,020 --> 00:07:10,270 that I used last time for starting and ending. 122 00:07:10,270 --> 00:07:11,880 All right, I think that is a 123 00:07:11,880 --> 00:07:14,080 trellis-oriented generator matrix. 124 00:07:14,080 --> 00:07:15,210 Well, it is. 125 00:07:15,210 --> 00:07:18,440 We found a matrix that has distinct starting times and 126 00:07:18,440 --> 00:07:19,690 ending times, so it is. 127 00:07:23,810 --> 00:07:26,125 Now what's its state branch complexity? 128 00:07:29,710 --> 00:07:35,320 The state dimension is 0. 129 00:07:35,320 --> 00:07:37,510 Now what are our spans now? 130 00:07:40,730 --> 00:07:45,190 Two are the same, but two are larger. 131 00:07:45,190 --> 00:07:49,370 OK, so now when we count, we get that the state complexity 132 00:07:49,370 --> 00:07:54,200 goes 0, 1, 2, 3, there are actually four that are now 133 00:07:54,200 --> 00:07:55,830 active in the center. 134 00:07:55,830 --> 00:08:00,430 3, 2, 1, 0. 135 00:08:00,430 --> 00:08:03,090 OK, and that's worse than we had before, at 136 00:08:03,090 --> 00:08:05,670 least in one place. 137 00:08:05,670 --> 00:08:11,320 If we look at the dimensions of the branch spaces, which 138 00:08:11,320 --> 00:08:16,020 I'm arguing is more fundamental, then we get 1, 2, 139 00:08:16,020 --> 00:08:19,730 3, again, 4, 4. 140 00:08:19,730 --> 00:08:26,470 So the branch complexity is now 16 rather than 8. 141 00:08:26,470 --> 00:08:28,395 It's going to be more difficult, definitely, to 142 00:08:28,395 --> 00:08:30,620 decode this code with the Viterbi algorithm. 143 00:08:30,620 --> 00:08:33,669 This is a more complex representation of an 144 00:08:33,669 --> 00:08:37,429 effectively equivalent code. 145 00:08:37,429 --> 00:08:40,140 Well, so is there-- 146 00:08:40,140 --> 00:08:42,740 so this is a bad permutation, it's something I 147 00:08:42,740 --> 00:08:44,960 didn't want to do. 148 00:08:44,960 --> 00:08:48,570 Is it possible there's some good permutation that would 149 00:08:48,570 --> 00:08:51,410 reduce the complexity? 150 00:08:51,410 --> 00:08:56,350 In general, when we're given a block code, traditionally at 151 00:08:56,350 --> 00:08:59,726 least, we think of it as the coordinates as just being not 152 00:08:59,726 --> 00:09:00,560 in any particular order. 153 00:09:00,560 --> 00:09:06,310 We can write down a generator matrix, but the view is that 154 00:09:06,310 --> 00:09:11,960 any permutation of the symbol times is perfectly legitimate. 155 00:09:11,960 --> 00:09:14,990 Certainly, if we send it over a memoryless channel, it's not 156 00:09:14,990 --> 00:09:16,890 going to affect the performance of the code, what 157 00:09:16,890 --> 00:09:18,340 order we send the symbols in. 158 00:09:18,340 --> 00:09:24,830 So we should certainly regard all such codes as equivalent. 159 00:09:24,830 --> 00:09:27,210 But we can affect the complexity, for better or for 160 00:09:27,210 --> 00:09:31,330 worse, so we should try to -- 161 00:09:31,330 --> 00:09:34,670 presumably, if we want a more fundamental complexity metric 162 00:09:34,670 --> 00:09:38,480 for the code, we want to find the complexity of the least 163 00:09:38,480 --> 00:09:43,240 complex trellis realization under all permutations of the 164 00:09:43,240 --> 00:09:44,490 coordinates. 165 00:09:47,210 --> 00:09:53,520 All right, so that's a good issue, but very little is 166 00:09:53,520 --> 00:09:56,295 known about this issue. 167 00:09:56,295 --> 00:10:00,240 It is known that finding the best permutation of an 168 00:10:00,240 --> 00:10:04,480 arbitrary linear code is an NP-hard problem. 169 00:10:04,480 --> 00:10:06,040 So this is going to be hard. 170 00:10:08,790 --> 00:10:13,330 We know for some specific classes of codes what the best 171 00:10:13,330 --> 00:10:16,000 permutation is. 172 00:10:16,000 --> 00:10:20,220 For instance, the Reed-Muller codes are nice because it's 173 00:10:20,220 --> 00:10:23,810 been proved by Kasami and others that the standard 174 00:10:23,810 --> 00:10:26,270 coordinate ordering, the one that I've been using all along 175 00:10:26,270 --> 00:10:30,487 based on this length doubling construction, the u-u plus v 176 00:10:30,487 --> 00:10:35,030 construction, gives you the minimal trellis complexity. 177 00:10:35,030 --> 00:10:37,580 All right, so we know what the best ordering is for the 178 00:10:37,580 --> 00:10:38,830 Reed-Muller codes. 179 00:10:42,580 --> 00:10:48,210 And in a minute, I'm going to prove what the best possible 180 00:10:48,210 --> 00:10:50,210 we can get bounds-- 181 00:10:50,210 --> 00:10:52,430 in particular, the Muder bounds, which I'm going to 182 00:10:52,430 --> 00:10:55,480 discuss next-- 183 00:10:55,480 --> 00:10:59,460 which bound the best possible complexity that 184 00:10:59,460 --> 00:11:01,040 you could ever get. 185 00:11:01,040 --> 00:11:04,350 And in some cases, we can find coordinate orderings that meet 186 00:11:04,350 --> 00:11:08,630 that bound, notably in the case of the 24-12-8 Golay 187 00:11:08,630 --> 00:11:12,180 code, we can find a coordinate ordering that's very closely 188 00:11:12,180 --> 00:11:15,295 related to the Reed-Muller ordering such that 189 00:11:15,295 --> 00:11:16,880 the bound is met. 190 00:11:16,880 --> 00:11:19,380 And so we know that we've found the best 191 00:11:19,380 --> 00:11:20,463 complexity for that code. 192 00:11:20,463 --> 00:11:22,590 It couldn't possibly be any better. 193 00:11:22,590 --> 00:11:27,200 But apart from certain very nice cases like those, we 194 00:11:27,200 --> 00:11:27,320 don't know. 195 00:11:27,320 --> 00:11:32,000 For instance, take an arbitrary BCH code. 196 00:11:32,000 --> 00:11:34,810 What's the best coordinate ordering for that? 197 00:11:34,810 --> 00:11:37,020 We don't know. 198 00:11:37,020 --> 00:11:43,880 So this is the open question in this subject, and it's now 199 00:11:43,880 --> 00:11:46,605 been open long enough and enough smart people have 200 00:11:46,605 --> 00:11:49,890 looked at it, and we've got this NP-hardness result, so 201 00:11:49,890 --> 00:11:53,070 that it looks like it's going to stay open indefinitely. 202 00:11:56,150 --> 00:12:01,200 All right, let me discuss, however, this Muder bound, 203 00:12:01,200 --> 00:12:05,040 because it's simple and sometimes definitive. 204 00:12:11,910 --> 00:12:12,165 Yes? 205 00:12:12,165 --> 00:12:13,415 AUDIENCE: [INAUDIBLE]? 206 00:12:16,100 --> 00:12:21,520 PROFESSOR: Minimal branch complexity or any other 207 00:12:21,520 --> 00:12:22,540 measure that you like. 208 00:12:22,540 --> 00:12:25,192 They all are quite fungible, it turns out. 209 00:12:31,310 --> 00:12:33,560 OK, The Muder bound. 210 00:12:37,260 --> 00:12:44,925 Let's look at a particular code and suppose we just know 211 00:12:44,925 --> 00:12:46,980 its parameters, n, k, d. 212 00:12:51,590 --> 00:12:56,750 Let's ask what the minimal trellis complexity could be 213 00:12:56,750 --> 00:13:01,740 for a particular coordinate ordering, say, that divides -- 214 00:13:01,740 --> 00:13:06,930 we have a certain number of past symbols, call it n_p, and 215 00:13:06,930 --> 00:13:11,690 a certain number of future symbols, n_f. 216 00:13:11,690 --> 00:13:15,740 And I'm going to ask what the minimum state complexity, or I 217 00:13:15,740 --> 00:13:17,370 could also ask what's the minimum 218 00:13:17,370 --> 00:13:21,170 branch complexity for-- 219 00:13:21,170 --> 00:13:24,220 let me do now a state complexity, going back to 220 00:13:24,220 --> 00:13:29,320 where we started from, for a a certain division in the past 221 00:13:29,320 --> 00:13:30,660 and future. 222 00:13:30,660 --> 00:13:38,825 To be concrete, let me take the 24-12-8 Golay code. 223 00:13:38,825 --> 00:13:43,570 It turns out that it's been proved that all codes with 224 00:13:43,570 --> 00:13:45,200 these parameters are equivalent. 225 00:13:45,200 --> 00:13:50,230 So we can just call this code the binary Golay code. 226 00:13:50,230 --> 00:13:57,240 And let's look for any division into 16 future 227 00:13:57,240 --> 00:14:00,420 coordinates and 8 past coordinates. 228 00:14:00,420 --> 00:14:02,980 So we don't know what the ordering is going to be, but 229 00:14:02,980 --> 00:14:04,790 we could say a few things. 230 00:14:04,790 --> 00:14:07,600 Remember how we get the state complexity. 231 00:14:07,600 --> 00:14:11,460 We look for what's the dimension of the past subcode. 232 00:14:11,460 --> 00:14:16,420 So here's a generator for the past subcode here, the code 233 00:14:16,420 --> 00:14:19,240 whose support is the past. 234 00:14:19,240 --> 00:14:23,370 We can ask what the dimension of the future subcode is, and 235 00:14:23,370 --> 00:14:28,035 then the dimension of the state space is the dimension 236 00:14:28,035 --> 00:14:30,100 of what's left. 237 00:14:30,100 --> 00:14:34,740 So it's dimension of c minus the dimension of the past 238 00:14:34,740 --> 00:14:40,380 subcode minus the dimension of the future subcode. 239 00:14:40,380 --> 00:14:43,220 State-space theorem. 240 00:14:43,220 --> 00:14:50,130 OK, but we know a few things about this code. 241 00:14:50,130 --> 00:14:53,440 We know what its length is and we know what its minimum 242 00:14:53,440 --> 00:14:54,990 distance is. 243 00:14:54,990 --> 00:15:00,340 So for this particular case, we have that c past is an 244 00:15:00,340 --> 00:15:08,450 8-something-8 code, because its minimum distance has got 245 00:15:08,450 --> 00:15:10,000 to be at least as great as this. 246 00:15:10,000 --> 00:15:13,670 It's a subcode of this code. 247 00:15:13,670 --> 00:15:18,370 So any non-zero code words in this past subcode have to have 248 00:15:18,370 --> 00:15:20,810 minimum weight of at least 8. 249 00:15:20,810 --> 00:15:22,660 Correct? 250 00:15:22,660 --> 00:15:24,530 All right. 251 00:15:24,530 --> 00:15:27,420 And similarly, the future subcode, in this 252 00:15:27,420 --> 00:15:30,590 case, is a 16-k -- 253 00:15:30,590 --> 00:15:32,100 let's call this past -- 254 00:15:32,100 --> 00:15:34,260 k-future-8 code. 255 00:15:37,230 --> 00:15:39,720 At least 8. 256 00:15:39,720 --> 00:15:44,660 So that imposes some bounds on the dimension of this code. 257 00:15:44,660 --> 00:15:49,610 How large could the dimension possibly be? 258 00:15:49,610 --> 00:15:51,075 All right, anybody? 259 00:15:51,075 --> 00:15:58,210 For the past subcode its clearly, we have that k_p less 260 00:15:58,210 --> 00:15:59,240 than or equal to 1. 261 00:15:59,240 --> 00:16:01,530 We could have, at most, one generator here. 262 00:16:01,530 --> 00:16:03,660 And if there were one, it would have to be 263 00:16:03,660 --> 00:16:05,990 the all-1 code word. 264 00:16:05,990 --> 00:16:09,000 Or the all-1 symbols here, because it has 265 00:16:09,000 --> 00:16:11,000 to have weight 8. 266 00:16:11,000 --> 00:16:12,230 So that's all we can get. 267 00:16:12,230 --> 00:16:14,440 For the future code-- 268 00:16:18,650 --> 00:16:21,560 well, I guess you don't know this, but you can guess that 269 00:16:21,560 --> 00:16:27,580 the Reed-Muller code with length 16 and minimum distance 270 00:16:27,580 --> 00:16:31,640 8 is the best there is, that distance 16, and that happens 271 00:16:31,640 --> 00:16:33,200 to be true. 272 00:16:33,200 --> 00:16:38,650 So k-future can't possibly be better than 5, which is the 273 00:16:38,650 --> 00:16:41,715 dimension of the Reed-Muller code with those parameters. 274 00:16:46,680 --> 00:16:51,470 For more general cases, you have to look up tables on 275 00:16:51,470 --> 00:16:53,730 bounds on the best codes that have ever been found. 276 00:16:53,730 --> 00:16:57,860 But certainly for code lengths less than a 100, by this time 277 00:16:57,860 --> 00:17:02,230 we pretty much know all the best binary codes. 278 00:17:02,230 --> 00:17:06,089 So we can look up in a table what's the best code of length 279 00:17:06,089 --> 00:17:08,000 16 and distance 8? 280 00:17:08,000 --> 00:17:12,440 Well, it has five information symbols, dimension five. 281 00:17:12,440 --> 00:17:15,920 So we couldn't have more than one dimension here and we 282 00:17:15,920 --> 00:17:20,394 couldn't have more than five dimensions here, so the 283 00:17:20,394 --> 00:17:23,599 dimensional code itself is 12. 284 00:17:23,599 --> 00:17:26,119 We have to say that the dimension state-space is now 285 00:17:26,119 --> 00:17:28,160 going to be greater than equal to-- 286 00:17:28,160 --> 00:17:32,760 this is k_p minus 1 minus 5-- 287 00:17:32,760 --> 00:17:35,060 so, that's 6. 288 00:17:35,060 --> 00:17:41,810 So there have to be at least 6 generators down here that have 289 00:17:41,810 --> 00:17:45,000 support neither wholly on the past nor wholly on the future 290 00:17:45,000 --> 00:17:49,610 and therefore have to be included in the state-space. 291 00:17:49,610 --> 00:17:51,370 All right, that's the idea of the Muder bound. 292 00:17:59,800 --> 00:18:03,390 Now, we could do this for every single state-space. 293 00:18:03,390 --> 00:18:06,960 We simply need to know what the dimension is of the best 294 00:18:06,960 --> 00:18:12,290 code of each length with minimum distance 8. 295 00:18:12,290 --> 00:18:17,520 And we can do a similar thing for every possible partition 296 00:18:17,520 --> 00:18:19,860 in the past and future. 297 00:18:19,860 --> 00:18:23,210 And that will give us a set of bounds, and we go through that 298 00:18:23,210 --> 00:18:26,616 exercise in the notes. 299 00:18:26,616 --> 00:18:32,670 And I can write it on the board, maybe for memory. 300 00:18:32,670 --> 00:18:37,950 And OK, that tells us what the best possible 301 00:18:37,950 --> 00:18:40,550 state profile is. 302 00:18:40,550 --> 00:18:55,510 Let me guess state dimension profile is 0, 1, 303 00:18:55,510 --> 00:19:00,706 2, 3, 4, 5, 6, 7. 304 00:19:00,706 --> 00:19:08,490 Then it goes down to 6, 7, 8, 9, then it goes down to 8. 305 00:19:08,490 --> 00:19:11,430 This is at the central state-space. 306 00:19:11,430 --> 00:19:17,060 This is for dividing it in the way that we just did. 307 00:19:17,060 --> 00:19:20,060 Let's do this here. 308 00:19:20,060 --> 00:19:27,870 For the central state-space, what's the best k_p for a 12, 309 00:19:27,870 --> 00:19:31,760 k_p, 8 code? 310 00:19:31,760 --> 00:19:33,790 Let's make the past and future both 12. 311 00:19:38,470 --> 00:19:40,620 Anyone want to guess what the best dimension is? 312 00:19:43,530 --> 00:19:45,280 Let's think about it. 313 00:19:45,280 --> 00:19:48,520 Let's think of a code that's going to have to have 314 00:19:48,520 --> 00:19:49,620 minimum weight 8. 315 00:19:49,620 --> 00:19:54,456 So let's start out with g1. 316 00:19:54,456 --> 00:20:00,140 And we'll just make that look like that. 317 00:20:00,140 --> 00:20:01,250 It's going to have length 12. 318 00:20:01,250 --> 00:20:04,910 It's going to have to have at least eight 1s, so let's make 319 00:20:04,910 --> 00:20:07,030 them like that. 320 00:20:07,030 --> 00:20:12,704 We want to add another 1 so that the total code has a 321 00:20:12,704 --> 00:20:15,350 minimum distance 8. 322 00:20:15,350 --> 00:20:18,250 Subject to up to permutations, there's only one 323 00:20:18,250 --> 00:20:18,865 thing it can be. 324 00:20:18,865 --> 00:20:23,080 We need another weight-8 generator, and it needs to 325 00:20:23,080 --> 00:20:26,150 differ from this in at least four places so that when we 326 00:20:26,150 --> 00:20:30,840 take the mod 2 sum of these two, it has distance 8. 327 00:20:30,840 --> 00:20:35,140 And we want g1 plus g2. 328 00:20:35,140 --> 00:20:38,600 This is really the only way it can be. 329 00:20:38,600 --> 00:20:39,900 Are you with me? 330 00:20:39,900 --> 00:20:44,030 We have four code words in this dimension 2 code. 331 00:20:44,030 --> 00:20:45,670 And if they're going to have minimum distance 8, they're 332 00:20:45,670 --> 00:20:47,460 going to have to look like this, subject to up to 333 00:20:47,460 --> 00:20:48,710 permutation. 334 00:20:50,540 --> 00:20:54,010 So from that little argument, we conclude that, in this 335 00:20:54,010 --> 00:20:56,410 case, this is less than or equal to 2. 336 00:20:56,410 --> 00:21:01,470 This is less than or equal to 2, the Muder bound becomes 337 00:21:01,470 --> 00:21:05,865 that the dimension of the state-space has 338 00:21:05,865 --> 00:21:07,115 to be at least 8. 339 00:21:10,880 --> 00:21:13,640 And so in the same way, and then it becomes symmetrical on 340 00:21:13,640 --> 00:21:14,370 the other side. 341 00:21:14,370 --> 00:21:17,410 So I'm pretty sure I get it right. 342 00:21:17,410 --> 00:21:20,800 And from memory, that's the best possible state dimension 343 00:21:20,800 --> 00:21:25,140 profile, dot dot dot. 344 00:21:25,140 --> 00:21:29,505 And we actually know a coordinate ordering for the 345 00:21:29,505 --> 00:21:33,500 Golay code, a generator matrix for the Golay code, that gives 346 00:21:33,500 --> 00:21:35,440 this state dimension profile. 347 00:21:35,440 --> 00:21:39,070 So from that, we can conclude that we know an optimum 348 00:21:39,070 --> 00:21:44,670 coordinate ordering for the Golay code. 349 00:21:44,670 --> 00:21:47,140 I just want you to get the essence of the argument here. 350 00:21:49,670 --> 00:21:50,920 You with me? 351 00:21:52,660 --> 00:21:56,270 For branch spaces, the argument is much the same. 352 00:21:56,270 --> 00:22:02,220 Remember in branch spaces, we basically -- 353 00:22:02,220 --> 00:22:05,790 let me say it in a slightly different way than we did 354 00:22:05,790 --> 00:22:10,460 before -- we take the past to be up to time k. 355 00:22:10,460 --> 00:22:13,460 And we have the symbol time, which is just one time in here 356 00:22:13,460 --> 00:22:15,900 for the actual symbol we're looking at. 357 00:22:15,900 --> 00:22:18,770 And the future is one unit shorter than it was for the 358 00:22:18,770 --> 00:22:20,070 state-space. 359 00:22:20,070 --> 00:22:24,380 So when we compute the branch dimension, the 360 00:22:24,380 --> 00:22:28,390 past goes up to k. 361 00:22:28,390 --> 00:22:32,570 So this is 0 and before k. 362 00:22:32,570 --> 00:22:36,780 And the future goes from k plus one up to n. 363 00:22:40,410 --> 00:22:44,690 So this is what the matrix looks like for this. 364 00:22:44,690 --> 00:22:51,030 And the effect over here is that the future subcode now is 365 00:22:51,030 --> 00:22:52,900 one unit shorter. 366 00:22:52,900 --> 00:22:55,050 So let's do the same calculation here. 367 00:22:58,720 --> 00:23:04,830 If we look at a time where we have 12 in the past, this one 368 00:23:04,830 --> 00:23:07,940 time, the 13th symbol time that we're actually looking at 369 00:23:07,940 --> 00:23:14,370 and then 11 more in the future, the past subcode, 370 00:23:14,370 --> 00:23:18,160 again, could have dimension up to 2 because it has length 12 371 00:23:18,160 --> 00:23:19,950 and minimum distance 8. 372 00:23:19,950 --> 00:23:25,500 The future subcode now has a shorter length and by this 373 00:23:25,500 --> 00:23:27,395 little argument I went through here. 374 00:23:27,395 --> 00:23:30,740 If we take out one of these coordinates, it's 375 00:23:30,740 --> 00:23:32,790 not going to work. 376 00:23:32,790 --> 00:23:37,210 So if we only have length 11, the future subcode is the 377 00:23:37,210 --> 00:23:38,380 dimension -- 378 00:23:38,380 --> 00:23:39,530 still can only be 1. 379 00:23:39,530 --> 00:23:43,890 We can't possibly add another second generator in there. 380 00:23:43,890 --> 00:23:51,790 And so the dimension of the branch space has to be vastly 381 00:23:51,790 --> 00:23:57,170 greater than or equal to 9, at this point. 382 00:23:57,170 --> 00:24:00,760 OK, so the Golay code has branch complexity of at least 383 00:24:00,760 --> 00:24:04,760 512, we conclude. 384 00:24:04,760 --> 00:24:10,060 And, in fact, this best possible branch dimension 385 00:24:10,060 --> 00:24:12,880 profile looks like this. 386 00:24:12,880 --> 00:24:15,240 The branches are in the middle here. 387 00:24:15,240 --> 00:24:19,800 They're 1, 2, 3, 4, 5, 6, 6. 388 00:24:19,800 --> 00:24:25,400 Basically we have generators which look like this. 389 00:24:25,400 --> 00:24:28,010 And then there's one that comes down, there are three 390 00:24:28,010 --> 00:24:31,090 more that goes up and one that comes down. 391 00:24:31,090 --> 00:24:41,125 And from that, we get 1, 2, 3, 4, 6, 7, 7, 8 -- 392 00:24:44,820 --> 00:24:48,290 that can't be right. 393 00:24:48,290 --> 00:24:53,290 Goes down 7, goes up, it's still 7, then it goes up to 8, 394 00:24:53,290 --> 00:25:00,330 9, and 9, 9, 9. 395 00:25:00,330 --> 00:25:01,660 Something like that. 396 00:25:01,660 --> 00:25:03,685 That one's probably wrong, and symmetrically. 397 00:25:06,470 --> 00:25:08,565 Where this 9 is the one we just calculated. 398 00:25:11,190 --> 00:25:15,850 Similarly, if you come down to the neighbor of this 6, it's 399 00:25:15,850 --> 00:25:18,320 got a merge on one side, a diverge on the other 400 00:25:18,320 --> 00:25:20,870 side, and that's 7. 401 00:25:20,870 --> 00:25:26,450 OK, so we conclude that the Golay code has branch 402 00:25:26,450 --> 00:25:29,090 dimension not less than 512 within 9. 403 00:25:29,090 --> 00:25:29,579 Yeah? 404 00:25:29,579 --> 00:25:36,106 AUDIENCE: Suppose we take past and future separated by 12, 12 405 00:25:36,106 --> 00:25:38,830 in the past, 12 in the future. 406 00:25:38,830 --> 00:25:41,620 Suppose we did it as they did in [UNINTELLIGIBLE] state? 407 00:25:41,620 --> 00:25:43,250 PROFESSOR: OK, but that's not the way we do branches. 408 00:25:43,250 --> 00:25:45,290 We've got to isolate the symbol time. 409 00:25:45,290 --> 00:25:49,030 AUDIENCE: Suppose we took 12-branch-12 and we calculate 410 00:25:49,030 --> 00:25:52,450 the state complexity, that's 8. 411 00:25:52,450 --> 00:25:59,435 And then we took 13 and 11, we get the state complexity 9. 412 00:26:03,530 --> 00:26:10,130 PROFESSOR: All right, you're saying if we do this, we 413 00:26:10,130 --> 00:26:12,670 should get the state complexity 9. 414 00:26:12,670 --> 00:26:14,840 And that's correct, we do. 415 00:26:14,840 --> 00:26:18,095 AUDIENCE: The maximum of those two numbers, is it the same as 416 00:26:18,095 --> 00:26:19,025 branch complexity? 417 00:26:19,025 --> 00:26:20,902 PROFESSOR: It turns out to be in this case. 418 00:26:20,902 --> 00:26:21,630 AUDIENCE: Not always? 419 00:26:21,630 --> 00:26:22,880 PROFESSOR: Not always. 420 00:26:28,290 --> 00:26:34,070 But if we go to 14, 10, then it turns out we can add 421 00:26:34,070 --> 00:26:39,370 another one here because we can add another generator that 422 00:26:39,370 --> 00:26:43,020 looks like something like this. 423 00:26:43,020 --> 00:26:50,660 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1. 424 00:26:50,660 --> 00:26:53,860 So if we have two more units, we can add another generator. 425 00:26:53,860 --> 00:26:56,380 Sort of Reed-Muller style. 426 00:26:56,380 --> 00:27:02,886 And so this goes up to 3 and this goes down to 8. 427 00:27:02,886 --> 00:27:05,460 AUDIENCE: So the branch complexity has to be at least 428 00:27:05,460 --> 00:27:07,020 the maximum of those two, right? 429 00:27:07,020 --> 00:27:08,270 PROFESSOR: Obviously, right. 430 00:27:12,085 --> 00:27:14,830 And really, together these give you the story of where 431 00:27:14,830 --> 00:27:17,820 the thing has to diverge and merge. 432 00:27:17,820 --> 00:27:20,720 In this case, this is a self-dual code, so you only do 433 00:27:20,720 --> 00:27:22,170 one at each time. 434 00:27:22,170 --> 00:27:24,090 So with this sort of thing, you can 435 00:27:24,090 --> 00:27:26,800 pretty well pin it down. 436 00:27:26,800 --> 00:27:31,500 Again, I don't know how far to take this lecture. 437 00:27:31,500 --> 00:27:33,900 I'm really just trying to get to you the idea of the Muder 438 00:27:33,900 --> 00:27:37,730 bound from this kind of argument, therefore we can get 439 00:27:37,730 --> 00:27:40,980 a bound on the best possible state dimension profile and 440 00:27:40,980 --> 00:27:45,080 the best possible branch complexity profile. 441 00:27:45,080 --> 00:27:52,840 If you compute this bound and you're able to find an actual 442 00:27:52,840 --> 00:27:55,000 generator matrix that meets this bound, you know you've 443 00:27:55,000 --> 00:27:56,990 found the best one. 444 00:27:56,990 --> 00:28:01,770 But this rarely happens, this is a very special code. 445 00:28:01,770 --> 00:28:05,230 It's not the way you prove the optimality of the longer 446 00:28:05,230 --> 00:28:07,990 Reed-Muller codes, they're not good enough to -- 447 00:28:07,990 --> 00:28:12,830 this really assumes that you can get the best possible 448 00:28:12,830 --> 00:28:15,190 sub-codes at every point. 449 00:28:15,190 --> 00:28:17,710 And you can imagine that's very hard to do. 450 00:28:17,710 --> 00:28:20,470 It would be a very special case where you can optimize 451 00:28:20,470 --> 00:28:24,655 this for every single possible cut between past and future. 452 00:28:27,520 --> 00:28:31,760 OK, so that's the Muder bound. 453 00:28:31,760 --> 00:28:37,460 And you have a homework exercise on that as well. 454 00:28:37,460 --> 00:28:41,900 Oh, there's another case where we can find what the best 455 00:28:41,900 --> 00:28:43,500 possible trellis complexity is. 456 00:28:43,500 --> 00:28:49,900 It's MDS codes over GFq, and you're going to do that on the 457 00:28:49,900 --> 00:28:51,580 homework as well. 458 00:28:51,580 --> 00:28:59,030 And there, we basically show that an MDS code, its n, k, d 459 00:28:59,030 --> 00:29:02,280 are such that it has to have the worst possible state 460 00:29:02,280 --> 00:29:06,470 complexity and branch complexity. 461 00:29:06,470 --> 00:29:09,960 So because it's such a tightly-packed code, in 462 00:29:09,960 --> 00:29:15,260 Hamming distance sense, it forces you to have the worst 463 00:29:15,260 --> 00:29:18,155 possible complexity that you could have. 464 00:29:18,155 --> 00:29:22,530 In other words, you have diverges immediately and as 465 00:29:22,530 --> 00:29:27,280 long as you can, and then they're all packed together on 466 00:29:27,280 --> 00:29:29,920 the left side and the merges are all packed together on the 467 00:29:29,920 --> 00:29:33,380 right side, and that's the only way it can be. 468 00:29:33,380 --> 00:29:35,675 I'm just speaking very impressionistically. 469 00:29:35,675 --> 00:29:37,865 But you'll see what I mean when you get to the homework, 470 00:29:37,865 --> 00:29:40,610 if you haven't done it already. 471 00:29:40,610 --> 00:29:48,970 OK, so we also know MDS codes, we know their trellis 472 00:29:48,970 --> 00:29:51,390 complexity and it's terrible. 473 00:29:51,390 --> 00:29:55,700 We can't get any effective reduction. 474 00:29:55,700 --> 00:29:58,860 Really, over exhaustive decoding methods with MDS 475 00:29:58,860 --> 00:30:03,040 codes, we find that the trellis complexity is like 476 00:30:03,040 --> 00:30:05,360 either q to the k or q to the n minus k, 477 00:30:05,360 --> 00:30:06,720 depending on which is less. 478 00:30:10,959 --> 00:30:13,205 So that's as bad as it could possibly be. 479 00:30:13,205 --> 00:30:19,790 So if the trellis doesn't help for MDS codes, in general 480 00:30:19,790 --> 00:30:20,930 cyclic codes -- 481 00:30:20,930 --> 00:30:23,940 which were subject to a lot of study -- if we put the code 482 00:30:23,940 --> 00:30:28,790 into cyclic form, which if you remember Reed-Solomon codes 483 00:30:28,790 --> 00:30:32,890 and BCH codes can always be put into cyclic form, at least 484 00:30:32,890 --> 00:30:35,050 the shortened versions of them. 485 00:30:35,050 --> 00:30:37,610 In that case, the generator matrix looks 486 00:30:37,610 --> 00:30:39,970 like this one up here. 487 00:30:39,970 --> 00:30:47,880 It looks like x, x, x, x, x, x, x, 0, 0, 0, 0, 0, 0, 0, x, 488 00:30:47,880 --> 00:30:53,820 x, x, x, x, x, x, 0, 0, x, x, x, x, x, x, x. 489 00:30:53,820 --> 00:30:56,830 In other words, the generators are just cyclic 490 00:30:56,830 --> 00:30:58,080 shifts of one another. 491 00:31:00,500 --> 00:31:07,300 Down to x, x, x, x, x, x, x, 0, 0, 0, 0. 492 00:31:10,570 --> 00:31:13,190 That is a generator matrix that is a trellis-oriented 493 00:31:13,190 --> 00:31:15,930 generator matrix because all the starting times are 494 00:31:15,930 --> 00:31:18,950 different and all the stopping times are different. 495 00:31:18,950 --> 00:31:22,080 So this is the trellis-oriented generator 496 00:31:22,080 --> 00:31:27,960 matrix for cyclic code, and it's in the coordinate 497 00:31:27,960 --> 00:31:30,440 ordering in which it is cyclic. 498 00:31:30,440 --> 00:31:36,210 And it's always going to have this worst case -- all the 499 00:31:36,210 --> 00:31:40,170 branch diverges are going to occur over here and you may 500 00:31:40,170 --> 00:31:41,390 get nothing for a while. 501 00:31:41,390 --> 00:31:47,330 And then all the merges over here, which if it's a rate 502 00:31:47,330 --> 00:31:51,230 less than 1/2 code, this means that the trellis complexity is 503 00:31:51,230 --> 00:31:55,070 basically going to be 2 to the k. 504 00:31:55,070 --> 00:31:57,320 Or if it's a high-rate code, it means it's going to be 2 to 505 00:31:57,320 --> 00:31:58,710 the n minus k. 506 00:31:58,710 --> 00:32:01,180 And again, you get no improvement over some 507 00:32:01,180 --> 00:32:04,410 exhaustive minimum distance decoding, or maximum 508 00:32:04,410 --> 00:32:06,770 likelihood decoding. 509 00:32:06,770 --> 00:32:10,240 So cyclic is very bad from a trellis point of view. 510 00:32:16,360 --> 00:32:19,110 So I think that's about all the general 511 00:32:19,110 --> 00:32:22,010 comments I can make. 512 00:32:22,010 --> 00:32:27,060 The very last thing that I want to mention is 513 00:32:27,060 --> 00:32:29,580 we now want to -- 514 00:32:29,580 --> 00:32:33,580 all right, so through this nice little theory, we've got 515 00:32:33,580 --> 00:32:40,480 a way of trellis decoding (maximum likelihood decoding), 516 00:32:40,480 --> 00:32:46,400 via the Viterbi algorithm, of block codes. 517 00:32:46,400 --> 00:32:52,600 And now, did we achieve any improvements in terms of 518 00:32:52,600 --> 00:33:00,360 performance versus complexity versus the convolutional codes 519 00:33:00,360 --> 00:33:03,390 that we looked at previously? 520 00:33:06,120 --> 00:33:12,040 And the basic answer is, no. 521 00:33:12,040 --> 00:33:17,850 If you compare the tables, 522 00:33:17,850 --> 00:33:19,970 convolutional are always better. 523 00:33:19,970 --> 00:33:22,590 I'll just say it. 524 00:33:22,590 --> 00:33:26,890 You can imagine that there have been long debates in the 525 00:33:26,890 --> 00:33:31,580 fraternity, and that certainly some of the motivation for 526 00:33:31,580 --> 00:33:34,390 trellis decoding of block codes was, oh, gee, now we 527 00:33:34,390 --> 00:33:37,710 have a good way to decode block codes, just like we did 528 00:33:37,710 --> 00:33:39,500 for convolutional codes. 529 00:33:39,500 --> 00:33:43,500 And maybe now we'll find that block codes are worthwhile. 530 00:33:43,500 --> 00:33:46,190 But the answer is they're not. 531 00:33:46,190 --> 00:33:50,840 I should put this comparison into Chapter 10. 532 00:33:50,840 --> 00:33:54,205 You can actually get this comparison by taking the table 533 00:33:54,205 --> 00:33:56,070 -- for instance, for Reed-Muller codes. 534 00:33:56,070 --> 00:33:59,010 In terms of performance versus complexity, Reed-Muller codes 535 00:33:59,010 --> 00:34:00,250 are very good. 536 00:34:00,250 --> 00:34:03,040 I would say as a general statement, they're the best 537 00:34:03,040 --> 00:34:07,470 block codes that we know of, in terms of performance versus 538 00:34:07,470 --> 00:34:11,679 trellis decoding complexity, because they have this nice 539 00:34:11,679 --> 00:34:16,080 structure that gives them very low-complexity trellises for 540 00:34:16,080 --> 00:34:19,820 their n, k, d parameters. 541 00:34:19,820 --> 00:34:22,909 So in this sense, it's a fair comparison. 542 00:34:22,909 --> 00:34:25,540 Reed-Muller codes are the best general class we know of in 543 00:34:25,540 --> 00:34:27,810 terms of trellis complexity. 544 00:34:27,810 --> 00:34:35,270 We compare them against binary convolutional codes and 545 00:34:35,270 --> 00:34:38,530 they're not nearly as good. 546 00:34:38,530 --> 00:34:40,330 I've already talked about the comparison. 547 00:34:40,330 --> 00:34:44,630 It seems like a very fair comparison to take our rate 548 00:34:44,630 --> 00:34:48,310 1/2 four-state convolutional code example, which had 549 00:34:48,310 --> 00:34:53,690 distance 5, which had not only a nominal coding gain of 4 dB, 550 00:34:53,690 --> 00:34:58,020 but it also has an effective coding gain of 4 dB. 551 00:34:58,020 --> 00:35:03,470 And for a block code, this 8, 4, 4 code is a very fair 552 00:35:03,470 --> 00:35:04,810 comparison. 553 00:35:04,810 --> 00:35:08,780 It's also rate 1/2, so it has the same spectral efficiency. 554 00:35:08,780 --> 00:35:12,770 It also has a four-state trellis that looks very much 555 00:35:12,770 --> 00:35:17,100 like a rate 1/2 convolutional code trellis. 556 00:35:17,100 --> 00:35:20,265 But it turns out it only has minimum distance of 4 rather 557 00:35:20,265 --> 00:35:27,720 than 5, so its nominal coding gain is only 3 dB 558 00:35:27,720 --> 00:35:28,850 rather than 4 dB. 559 00:35:28,850 --> 00:35:32,250 Its effective coding gain is less. 560 00:35:32,250 --> 00:35:37,400 Its effective coding gain is 2.6 dN because it has 14 561 00:35:37,400 --> 00:35:41,030 nearest neighbors, which is a 3 1/2 nearest neighbors per 562 00:35:41,030 --> 00:35:45,550 bit, and that costs you about 4/10 of a dN. 563 00:35:45,550 --> 00:35:50,340 And so really, you compare the performance of the 564 00:35:50,340 --> 00:35:53,040 convolutional to the block code, that's very much better. 565 00:35:56,620 --> 00:36:00,330 Another head-to-head comparison might be between 566 00:36:00,330 --> 00:36:04,380 the 32, 16, 8 Reed-Muller code. 567 00:36:04,380 --> 00:36:06,680 That's a rate 1/2 code. 568 00:36:06,680 --> 00:36:12,030 It's a very good one, it's known to be the best with 569 00:36:12,030 --> 00:36:14,460 those parameters, n, k, d. 570 00:36:14,460 --> 00:36:16,670 So what's its nominal coding gain? 571 00:36:16,670 --> 00:36:17,070 Anyone? 572 00:36:17,070 --> 00:36:18,860 It has rate 1/2, minimum distance 8, its 573 00:36:18,860 --> 00:36:22,410 nominal coding gain is? 574 00:36:22,410 --> 00:36:24,570 Hello? 575 00:36:24,570 --> 00:36:26,460 6 dB. 576 00:36:26,460 --> 00:36:29,100 You take the rate times the minimum distance-- 577 00:36:29,100 --> 00:36:31,210 1/2 times 8, you get 4. 578 00:36:31,210 --> 00:36:32,890 So that's the nominal coding gain. 579 00:36:32,890 --> 00:36:38,030 A factor of 4, which in dB terms is 6 dB. 580 00:36:38,030 --> 00:36:41,510 But again, it has lots of minimum-weight code words. 581 00:36:41,510 --> 00:36:45,850 It has 620 minimum-weight code words. 582 00:36:45,850 --> 00:36:48,760 And therefore, the number of minimum-weight code words per 583 00:36:48,760 --> 00:36:54,690 bit is 39, which is between five and six factors of 3, so 584 00:36:54,690 --> 00:37:01,220 it costs you 5 and 1/2 times 0.2 dB in terms of effective 585 00:37:01,220 --> 00:37:03,100 coding gain, which brings you down to an effective coding 586 00:37:03,100 --> 00:37:04,350 gain of 4.9 dB. 587 00:37:06,700 --> 00:37:08,142 I assume you're all very fluent in 588 00:37:08,142 --> 00:37:11,970 doing these, as I am. 589 00:37:11,970 --> 00:37:15,420 And what's its complexity? 590 00:37:15,420 --> 00:37:21,770 If you look its state complexity, let's take-- 591 00:37:21,770 --> 00:37:24,320 we're looking at the 32, 16, 8. 592 00:37:24,320 --> 00:37:29,960 And all the Reed-Muller codes have four section trellises, 593 00:37:29,960 --> 00:37:32,040 you proved that this week. 594 00:37:32,040 --> 00:37:36,350 If you just measure the state complexity of four places, 595 00:37:36,350 --> 00:37:40,130 equally spaced with four equal sections, then all the 596 00:37:40,130 --> 00:37:44,800 trellises wind up looking like this, for a general 597 00:37:44,800 --> 00:37:47,390 Reed-Muller trellis -- 598 00:37:47,390 --> 00:37:48,640 a four-section trellis. 599 00:37:53,400 --> 00:37:57,250 And for the 32, 16, 8, it turns out you get eight 600 00:37:57,250 --> 00:38:02,570 parallel sections like that with eight of these each. 601 00:38:02,570 --> 00:38:05,875 In any case, it's 64 states at each of these boundaries. 602 00:38:09,830 --> 00:38:12,600 So you could say it's a 64-state trellis. 603 00:38:12,600 --> 00:38:16,370 But if you're more honest, just exactly as with the Golay 604 00:38:16,370 --> 00:38:20,890 code calculation we went through, you find that the 605 00:38:20,890 --> 00:38:25,480 maximum branch complexity is 512. 606 00:38:25,480 --> 00:38:36,710 So branch complexity is 512. 607 00:38:36,710 --> 00:38:43,440 OK, and its nominal coding gain we said is 6 dB and its 608 00:38:43,440 --> 00:38:47,450 effective coding gain is 4.9 dB. 609 00:38:51,150 --> 00:38:56,065 OK, so let's look for a convolutional code of rate 1/2 610 00:38:56,065 --> 00:38:59,620 that has a coding gain of about 5 or 6 dB. 611 00:39:02,320 --> 00:39:05,010 And how far do we have to go? 612 00:39:05,010 --> 00:39:08,625 We actually only have to go up to a-- 613 00:39:08,625 --> 00:39:12,010 well, a 32-state code has an effective 614 00:39:12,010 --> 00:39:13,520 coding gain of 5.6 dB. 615 00:39:13,520 --> 00:39:19,300 A 64-state code we can get up to an effective coding gain of 616 00:39:19,300 --> 00:39:22,060 about 6 dB. 617 00:39:22,060 --> 00:39:26,700 In fact, the 64-state rate 1/2 convolutional code was a 618 00:39:26,700 --> 00:39:31,680 standard for a very long time, first in space communications 619 00:39:31,680 --> 00:39:33,810 and then in much more general applications. 620 00:39:33,810 --> 00:39:37,650 This was the code that Linkabit Corporation -- 621 00:39:37,650 --> 00:39:39,280 Jacobs and Viterbi -- 622 00:39:39,280 --> 00:39:43,730 rode to fame and fortune, made their first pile before they 623 00:39:43,730 --> 00:39:47,520 went on to found Qualcomm, where they made 624 00:39:47,520 --> 00:39:50,850 a much bigger pile. 625 00:39:50,850 --> 00:39:53,530 And so this was their-- 626 00:39:53,530 --> 00:39:55,936 they call it k equals 7. 627 00:39:55,936 --> 00:40:01,120 But in my terms, nu equals 6, a 64-state rate 1/2 628 00:40:01,120 --> 00:40:02,170 convolutional code. 629 00:40:02,170 --> 00:40:12,170 So let's take that rate 1/2 64-state convolutional code, 630 00:40:12,170 --> 00:40:14,500 just with the standard trellis. 631 00:40:14,500 --> 00:40:23,945 So its branch complexity is 128 at every time. 632 00:40:23,945 --> 00:40:27,920 It has a more regular structure than this guy. 633 00:40:27,920 --> 00:40:33,860 Its nominal coding gain is -- 634 00:40:33,860 --> 00:40:37,040 there are actually two of them, and I don't remember. 635 00:40:37,040 --> 00:40:40,210 There's one of them that has a nominal coding gain of 7, the 636 00:40:40,210 --> 00:40:41,030 other one is 6.5. 637 00:40:41,030 --> 00:40:45,210 So it's at least 6.5 dB. 638 00:40:45,210 --> 00:40:51,030 And its effective coding gain is still 6.1 dB and it's 639 00:40:51,030 --> 00:40:52,390 clearly much better than this. 640 00:40:57,900 --> 00:41:03,650 If you kind of try to plot a graph of this, you'll see that 641 00:41:03,650 --> 00:41:07,220 the effective coding gains of convolutional codes with the 642 00:41:07,220 --> 00:41:11,950 same state complexity is at least one dB better, as a 643 00:41:11,950 --> 00:41:14,910 general trend, than that of block codes. 644 00:41:14,910 --> 00:41:21,180 And of course, if you do it in terms of branch complexity, 645 00:41:21,180 --> 00:41:24,160 it's an even better comparison. 646 00:41:24,160 --> 00:41:31,480 So what happened was there was about a decade of effort on 647 00:41:31,480 --> 00:41:35,150 basically finding good trellis representations of block 648 00:41:35,150 --> 00:41:38,690 codes, and the net of it was that they couldn't get 649 00:41:38,690 --> 00:41:42,100 anywhere close to finding anything as good as 650 00:41:42,100 --> 00:41:43,100 convolutional codes. 651 00:41:43,100 --> 00:41:46,910 Although, it's a nice little piece of theory. 652 00:41:46,910 --> 00:41:49,460 And sometimes you want to block codes for other reasons. 653 00:41:49,460 --> 00:41:52,490 Sometimes you want short blocks. 654 00:41:52,490 --> 00:41:57,100 Sometimes you're limited to a block of less than 100. 655 00:41:57,100 --> 00:42:00,040 And in that case, you're certainly not going to do 656 00:42:00,040 --> 00:42:03,160 better in terms of n, k, d than the best block code of a 657 00:42:03,160 --> 00:42:04,770 length 100. 658 00:42:04,770 --> 00:42:06,700 Why are convolutional codes better? 659 00:42:06,700 --> 00:42:11,610 First of all, they obviously naturally are well-adapted to 660 00:42:11,610 --> 00:42:12,840 the trellis idea. 661 00:42:12,840 --> 00:42:20,300 They are linear time invariant machines that just have very 662 00:42:20,300 --> 00:42:22,550 nice, regular trellises. 663 00:42:22,550 --> 00:42:27,170 So they're naturally adapted. 664 00:42:27,170 --> 00:42:31,740 But also, they have exactly the right-- 665 00:42:31,740 --> 00:42:35,580 a rate 1/2 code is diverge, merge, diverge, merge, 666 00:42:35,580 --> 00:42:38,630 diverge, merge forever. 667 00:42:38,630 --> 00:42:41,270 If you want to make it into a block code, you can terminate 668 00:42:41,270 --> 00:42:43,980 it or you can tail-bite it, which is something we're going 669 00:42:43,980 --> 00:42:46,970 to talk about shortly. 670 00:42:46,970 --> 00:42:50,150 But whichever way you do it, to make it into a block code, 671 00:42:50,150 --> 00:42:50,780 you lose something. 672 00:42:50,780 --> 00:42:54,040 The termination, you don't lose minimum distance, but you 673 00:42:54,040 --> 00:42:54,820 do lose rate. 674 00:42:54,820 --> 00:42:58,090 You have to put in some dummy information symbols, so the 675 00:42:58,090 --> 00:43:02,250 rate becomes less than 1/2 if you terminate this code. 676 00:43:02,250 --> 00:43:05,360 Nonetheless, as I believe I commented earlier, that's a 677 00:43:05,360 --> 00:43:09,620 good way to come up with good block codes, is to terminate a 678 00:43:09,620 --> 00:43:12,140 good convolutional code. 679 00:43:12,140 --> 00:43:14,270 If you choose the parameters right, you can come up with 680 00:43:14,270 --> 00:43:16,350 pretty good codes. 681 00:43:16,350 --> 00:43:19,660 But nonetheless, in terms of block code parameters, you're 682 00:43:19,660 --> 00:43:23,650 not going to do better than what 50 years of block code, 683 00:43:23,650 --> 00:43:25,580 algebraic coding theory has come up with. 684 00:43:25,580 --> 00:43:30,240 So for short block codes with length less than 100, really 685 00:43:30,240 --> 00:43:34,100 the best we know of right now-- let's say 128-- 686 00:43:34,100 --> 00:43:37,890 is to pick a good Reed-Muller code or BCH code, find its 687 00:43:37,890 --> 00:43:41,640 best trellis and do maximum likelihood decoding. 688 00:43:46,010 --> 00:43:49,110 It used to be that you thought of 64 states as a lot of 689 00:43:49,110 --> 00:43:54,740 states, but clearly, technology has progressed. 690 00:43:54,740 --> 00:43:59,480 And now you wouldn't blink at 1,000 states or maybe even 691 00:43:59,480 --> 00:44:02,065 8,000 or 16,000 states. 692 00:44:02,065 --> 00:44:07,205 The biggest Viterbi decoder that was ever built had 2 to 693 00:44:07,205 --> 00:44:10,860 the 14 states for a space program out of JPL. 694 00:44:14,970 --> 00:44:17,460 So anyway, it's doable now. 695 00:44:22,860 --> 00:44:26,690 If you needed the best possible performance and you 696 00:44:26,690 --> 00:44:28,900 were constrained in block length, you'd pick the best 697 00:44:28,900 --> 00:44:32,850 block code at the rate that you wanted, and the best 698 00:44:32,850 --> 00:44:35,980 minimum distance at that rate, and the best effective coding 699 00:44:35,980 --> 00:44:38,500 gain, and you'd do maximum likelihood decoding with 700 00:44:38,500 --> 00:44:40,560 trellis decoding. 701 00:44:40,560 --> 00:44:43,020 But if that's not usually the situation -- 702 00:44:43,020 --> 00:44:45,470 the situation in data communications, because 703 00:44:45,470 --> 00:44:50,230 usually more you have a stream of data or a packet length of 704 00:44:50,230 --> 00:44:54,605 at least 1,000, and then it would certainly be better to 705 00:44:54,605 --> 00:44:55,855 use convolutional codes. 706 00:44:58,220 --> 00:44:59,850 And trellis decoding is -- 707 00:44:59,850 --> 00:45:01,860 as long as we're in this class of techniques. 708 00:45:01,860 --> 00:45:05,340 Where we're finally going in this course is to the new 709 00:45:05,340 --> 00:45:08,140 codes that have been developed in the past decade, 710 00:45:08,140 --> 00:45:12,090 capacity-approaching codes, turbo codes, low-density 711 00:45:12,090 --> 00:45:13,340 parity-check codes. 712 00:45:15,640 --> 00:45:19,230 These are simple enough to do so that as soon as you get 713 00:45:19,230 --> 00:45:24,250 past the small codes, length less than 100, you would 714 00:45:24,250 --> 00:45:25,910 switch over to that track. 715 00:45:25,910 --> 00:45:28,872 You would use a capacity-approaching code. 716 00:45:28,872 --> 00:45:33,080 But we don't know about them yet. 717 00:45:33,080 --> 00:45:36,010 I guess that's the end of my lecture on this, so it's a 718 00:45:36,010 --> 00:45:40,130 very good place for questions about state-of-the-art, where 719 00:45:40,130 --> 00:45:41,400 we've been, where we're going. 720 00:45:45,820 --> 00:45:47,560 The floor is always open. 721 00:45:47,560 --> 00:45:52,110 But that was all I was going to say about Chapter 10. 722 00:45:58,340 --> 00:46:03,040 Chapter 11, Codes on Graphs. 723 00:46:03,040 --> 00:46:09,170 And now we begin our final stretch drive towards 724 00:46:09,170 --> 00:46:11,960 capacity-approaching codes, which is going to be the 725 00:46:11,960 --> 00:46:15,320 subject of the next three chapters. 726 00:46:15,320 --> 00:46:19,890 Chapter 11 is sort of the beginnings of codes on graphs. 727 00:46:19,890 --> 00:46:25,940 We're going to look at some elementary representations. 728 00:46:25,940 --> 00:46:27,960 One of the ones we're going to look at is a trellis 729 00:46:27,960 --> 00:46:28,840 representation. 730 00:46:28,840 --> 00:46:30,700 That's one of the reasons we've been going through this 731 00:46:30,700 --> 00:46:39,330 theory, is we're going to, again, encounter trellises as 732 00:46:39,330 --> 00:46:46,900 a natural, simple, cycle-free, graphical representation that, 733 00:46:46,900 --> 00:46:49,830 in some sense, is about as good as you can ever do, as 734 00:46:49,830 --> 00:46:51,815 long as you don't allow cycles in graphs. 735 00:46:55,500 --> 00:46:59,110 But then we're going to go on to graphs with cycles, which 736 00:46:59,110 --> 00:47:01,320 is really the way to understand these 737 00:47:01,320 --> 00:47:03,580 capacity-approaching codes. 738 00:47:03,580 --> 00:47:06,370 So in this chapter, we'll start the subject, we'll 739 00:47:06,370 --> 00:47:09,000 introduce graphical representations of codes in 740 00:47:09,000 --> 00:47:12,141 more general terms. 741 00:47:12,141 --> 00:47:16,120 In Chapter 12, we'll talk about the generic decoding 742 00:47:16,120 --> 00:47:22,510 algorithms for codes on graphs called sum-product and min-sum 743 00:47:22,510 --> 00:47:25,720 decoding algorithms. 744 00:47:25,720 --> 00:47:31,830 And then in Chapter 13, we'll talk about the actual classes 745 00:47:31,830 --> 00:47:35,800 of capacity-approaching codes that people have developed, 746 00:47:35,800 --> 00:47:38,670 what their graphs are, how you decode them -- 747 00:47:38,670 --> 00:47:42,760 I hope in enough detail so that you'll get a good sense 748 00:47:42,760 --> 00:47:46,530 for how all this works. 749 00:47:46,530 --> 00:47:48,260 So that's where we're going. 750 00:47:56,130 --> 00:47:59,100 What are we talking about when we're talking 751 00:47:59,100 --> 00:48:00,420 about codes on graphs? 752 00:48:03,460 --> 00:48:08,760 There are many styles of representations of codes, and 753 00:48:08,760 --> 00:48:14,580 at this point, for linear codes, we have seen a number 754 00:48:14,580 --> 00:48:17,990 of ways we can characterize a code. 755 00:48:17,990 --> 00:48:21,130 One of the ways is just by giving a set of generators for 756 00:48:21,130 --> 00:48:23,880 the code, k generators for an n, k code. 757 00:48:23,880 --> 00:48:26,390 And that sort of characterizes the code. 758 00:48:26,390 --> 00:48:30,270 Another way is to basically give the generator matrix for 759 00:48:30,270 --> 00:48:34,880 the dual code, the h matrix, which becomes the parity-check 760 00:48:34,880 --> 00:48:37,030 matrix for the code. 761 00:48:37,030 --> 00:48:42,180 So we can also characterize the code by a 762 00:48:42,180 --> 00:48:43,540 dual-generator matrix. 763 00:48:43,540 --> 00:48:45,860 That's a particularly efficient way to do it if you 764 00:48:45,860 --> 00:48:50,190 have a high-rate code like a single parity-check code, it's 765 00:48:50,190 --> 00:48:55,450 better to say that the dual code is the repetition code 766 00:48:55,450 --> 00:49:00,160 and it's the set of all code words that are orthogonal to 767 00:49:00,160 --> 00:49:02,600 the all-one code word. 768 00:49:02,600 --> 00:49:05,745 That's the simplest characterization of the single 769 00:49:05,745 --> 00:49:06,380 parity-check code. 770 00:49:06,380 --> 00:49:10,962 In other words, that the sum of all bits is equal to 0. 771 00:49:10,962 --> 00:49:13,680 That's what orthogonal to the all-one code word means. 772 00:49:13,680 --> 00:49:16,920 So that's a simpler representation than giving a 773 00:49:16,920 --> 00:49:22,030 generator matrix formed for the single parity-check code. 774 00:49:22,030 --> 00:49:28,190 Or now we have this trellis representation, which 775 00:49:28,190 --> 00:49:30,870 certainly looks like a more graphical representation. 776 00:49:30,870 --> 00:49:35,150 It certainly characterizes the code in a very direct way. 777 00:49:35,150 --> 00:49:36,120 What does a trellis do? 778 00:49:36,120 --> 00:49:40,260 It basically displays all the code words on a graph. 779 00:49:40,260 --> 00:49:43,733 There's a 1:1 correspondence between the set of all paths, 780 00:49:43,733 --> 00:49:48,840 from the root to the n-node in the graph and the code. 781 00:49:48,840 --> 00:49:51,020 So that's another way to characterize the code. 782 00:49:53,650 --> 00:49:56,960 So now we're going to go to higher level graphical 783 00:49:56,960 --> 00:49:58,210 representations. 784 00:50:00,330 --> 00:50:09,910 As a first step, we talk about behavioral realizations. 785 00:50:09,910 --> 00:50:15,790 Again, a term from system theory, closely identified 786 00:50:15,790 --> 00:50:24,310 with Jan Willems, who promoted this way of representing 787 00:50:24,310 --> 00:50:26,400 linear systems. 788 00:50:26,400 --> 00:50:34,980 In linear system theory terms, the basic idea in behavioral 789 00:50:34,980 --> 00:50:40,030 systems theory is you describe a system by the set of all its 790 00:50:40,030 --> 00:50:41,530 possible trajectories. 791 00:50:41,530 --> 00:50:45,230 What are all the things it can do? 792 00:50:45,230 --> 00:50:48,180 And now, how do you characterize the trajectories? 793 00:50:48,180 --> 00:50:54,212 Often, you characterize them by a set of local constraints. 794 00:50:54,212 --> 00:50:59,170 You can think of it as equations that the system has 795 00:50:59,170 --> 00:51:01,900 to satisfy. 796 00:51:01,900 --> 00:51:05,110 Ordinarily, they're differential equations or 797 00:51:05,110 --> 00:51:06,960 partial differential equations or something. 798 00:51:06,960 --> 00:51:12,190 And if it satisfies all of those constraints, then it's a 799 00:51:12,190 --> 00:51:17,060 trajectory that satisfies all these partial differential 800 00:51:17,060 --> 00:51:18,830 equations, is a valid trajectory. 801 00:51:23,356 --> 00:51:25,530 So you can set up the whole system that way. 802 00:51:25,530 --> 00:51:35,525 So it characterizes a system by its trajectories. 803 00:51:38,170 --> 00:51:41,300 That's the fundamental thing. 804 00:51:41,300 --> 00:51:43,980 And it characterizes trajectories -- 805 00:51:43,980 --> 00:51:45,520 legitimate, valid trajectories -- 806 00:51:49,610 --> 00:51:50,860 by local constraints. 807 00:51:57,920 --> 00:52:00,350 Let me get a little bit more concrete. 808 00:52:00,350 --> 00:52:07,510 In coding terms, by trajectories we 809 00:52:07,510 --> 00:52:10,280 just mean code words. 810 00:52:10,280 --> 00:52:13,670 So this is a very compatible notion with the way we've been 811 00:52:13,670 --> 00:52:14,600 talking in coding. 812 00:52:14,600 --> 00:52:15,490 What is a code? 813 00:52:15,490 --> 00:52:18,480 It's simply a set of code words. 814 00:52:18,480 --> 00:52:21,900 So if we simply say, what are all the possible code words? 815 00:52:21,900 --> 00:52:23,870 We have to find all the trajectories of the code, and 816 00:52:23,870 --> 00:52:27,700 we have a very explicit example of that in trellises. 817 00:52:27,700 --> 00:52:29,120 We've called the paths 818 00:52:29,120 --> 00:52:32,750 trajectories through a trellis. 819 00:52:32,750 --> 00:52:33,680 What is a code? 820 00:52:33,680 --> 00:52:38,570 It's simply the set of all valid code words. 821 00:52:38,570 --> 00:52:43,640 And how do we characterize the valid code words? 822 00:52:43,640 --> 00:52:46,630 Well, in each of these styles -- 823 00:52:46,630 --> 00:52:50,850 the generator, parity-check, trellis style -- 824 00:52:50,850 --> 00:52:53,820 the code words are the code words that satisfy certain 825 00:52:53,820 --> 00:52:56,940 equations or constraints. 826 00:52:56,940 --> 00:52:59,756 For instance, the generator representation -- 827 00:53:04,050 --> 00:53:12,515 we've said a code is the set of all uG, such that u is in a 828 00:53:12,515 --> 00:53:14,930 field to the k. 829 00:53:14,930 --> 00:53:22,930 That's a generator matrix style of representing a code. 830 00:53:22,930 --> 00:53:29,580 In other words, we can say it's the set of all y, such 831 00:53:29,580 --> 00:53:41,210 that y equals uG for some u in Fk. 832 00:53:45,680 --> 00:53:55,480 Or we can say it's all y, such that y minus uG equals 0 for 833 00:53:55,480 --> 00:54:01,040 some u in Fk. 834 00:54:01,040 --> 00:54:08,480 So it's the set of all y that, together with some auxiliary 835 00:54:08,480 --> 00:54:11,130 variables which are not part of the code word -- 836 00:54:11,130 --> 00:54:14,450 here they're input variables, if you like, or information 837 00:54:14,450 --> 00:54:17,470 bits, information symbols. 838 00:54:17,470 --> 00:54:20,090 We'll come to think of these as state variables because 839 00:54:20,090 --> 00:54:22,720 they're hidden, they're not visible, they're just an 840 00:54:22,720 --> 00:54:24,125 auxiliary part of the description. 841 00:54:28,050 --> 00:54:30,570 They're code words that, together with some auxiliary 842 00:54:30,570 --> 00:54:37,710 variables that, in this case, can be freely chosen, satisfy 843 00:54:37,710 --> 00:54:40,800 a certain set of linear homogeneous equations. 844 00:54:45,810 --> 00:54:49,935 Something that's even better in the behavioral style is a 845 00:54:49,935 --> 00:54:51,822 parity-check representation. 846 00:54:51,822 --> 00:54:53,244 Let me put that up. 847 00:54:56,570 --> 00:54:58,820 In a parity-check representation, we say the 848 00:54:58,820 --> 00:55:08,980 code is the set of all y, such that y H transpose equals 0, 849 00:55:08,980 --> 00:55:12,495 where H is the generator matrix of the dual code. 850 00:55:15,190 --> 00:55:18,380 This is sometimes called a kernel representation. 851 00:55:18,380 --> 00:55:20,695 y is in the kernel of this linear transformation. 852 00:55:24,280 --> 00:55:28,240 We see H transpose is a linear transformation from n-tuples 853 00:55:28,240 --> 00:55:33,670 to n minus k-tuples, and the ones whose image is 0 under 854 00:55:33,670 --> 00:55:38,520 this transformation, the ones that map to 0 are legitimate 855 00:55:38,520 --> 00:55:39,590 code words. 856 00:55:39,590 --> 00:55:42,110 So this is a kernel representation, this is an 857 00:55:42,110 --> 00:55:44,810 image representation. 858 00:55:44,810 --> 00:55:47,170 Here, we view the code as the image of the linear 859 00:55:47,170 --> 00:55:53,160 transformation, which is characterized by g, where the 860 00:55:53,160 --> 00:55:55,500 inputs are free. 861 00:55:55,500 --> 00:55:58,520 But the parity-check representation is much more 862 00:55:58,520 --> 00:56:01,180 explicitly, we don't need any auxiliary variables here. 863 00:56:01,180 --> 00:56:04,580 It's simply the set of n-tuples that satisfy a 864 00:56:04,580 --> 00:56:06,655 certain set of linear homogeneous equations. 865 00:56:10,110 --> 00:56:14,890 So this maybe is a place where we start and, of course, it's 866 00:56:14,890 --> 00:56:16,840 the place where Bob Gallager started when he did 867 00:56:16,840 --> 00:56:20,050 low-density parity-check codes, which are a principal 868 00:56:20,050 --> 00:56:22,800 class of capacity-approaching codes. 869 00:56:26,300 --> 00:56:30,020 Then in either case, we characterize the code by, in 870 00:56:30,020 --> 00:56:33,410 one case, directly satisfying some equations. 871 00:56:33,410 --> 00:56:38,340 In the other case, here we say that the total behavior is the 872 00:56:38,340 --> 00:56:45,550 set of all y and u, such that y minus uG equals 0. 873 00:56:49,350 --> 00:56:54,640 So we're going to take the set of all n-tuples and k-tuples, 874 00:56:54,640 --> 00:56:56,870 such that y minus uG equals 0. 875 00:56:56,870 --> 00:57:02,250 And we say that's the total behavior of the code. 876 00:57:02,250 --> 00:57:07,910 The set of all y's and u's that satisfy this equation is 877 00:57:07,910 --> 00:57:08,840 the total behavior. 878 00:57:08,840 --> 00:57:12,160 And then we'll say the code word is just the projection of 879 00:57:12,160 --> 00:57:13,450 the behavior on the y's. 880 00:57:16,450 --> 00:57:18,870 In other words, it's the set of all y's, such that this is 881 00:57:18,870 --> 00:57:22,410 satisfied first some u. 882 00:57:22,410 --> 00:57:23,660 And that's an equivalent description. 883 00:57:27,420 --> 00:57:38,800 So the general setup, going back and forth, is that we 884 00:57:38,800 --> 00:57:42,945 have the observed symbols. 885 00:57:48,520 --> 00:57:51,970 And in the coding case, this will always just be the n 886 00:57:51,970 --> 00:57:54,420 symbols in the code word. 887 00:57:59,720 --> 00:58:03,340 These are the ones that we really care about. 888 00:58:03,340 --> 00:58:07,140 But we're also going to have some hidden or auxiliary 889 00:58:07,140 --> 00:58:16,000 symbols or state variables, hidden variables, which, in 890 00:58:16,000 --> 00:58:19,200 this case, for instance, are the u's. 891 00:58:19,200 --> 00:58:27,160 And we can introduce these at our convenience, just to make 892 00:58:27,160 --> 00:58:28,410 a good realization. 893 00:58:31,930 --> 00:58:36,820 So we have two types of symbols: the observed, or 894 00:58:36,820 --> 00:58:38,400 external variables-- 895 00:58:38,400 --> 00:58:39,680 sometimes these are called external 896 00:58:39,680 --> 00:58:42,730 variables, internal variables. 897 00:58:42,730 --> 00:58:44,430 Again, in system theory, you're 898 00:58:44,430 --> 00:58:46,760 consistently seeing this. 899 00:58:46,760 --> 00:58:48,910 There are some that we can actually 900 00:58:48,910 --> 00:58:50,720 observe, interact with. 901 00:58:50,720 --> 00:58:53,540 These are the observed symbols, and then there are 902 00:58:53,540 --> 00:58:56,550 internal variables to the system, very often called 903 00:58:56,550 --> 00:59:01,960 state variables, which we can't directly see but which 904 00:59:01,960 --> 00:59:06,340 are part of the description of the system, as in this case. 905 00:59:06,340 --> 00:59:11,100 The whole system, we find the set of all internal and 906 00:59:11,100 --> 00:59:13,735 external variables that satisfy a set of constraints. 907 00:59:19,140 --> 00:59:24,935 But we only care about the patterns of observed variables 908 00:59:24,935 --> 00:59:31,040 that are consistent with some set of internal variables. 909 00:59:31,040 --> 00:59:35,440 So the final element here in this general setup is 910 00:59:35,440 --> 00:59:44,370 constraints, which in this case are simply linear 911 00:59:44,370 --> 00:59:55,420 homogeneous equations on subsets of variables. 912 01:00:07,250 --> 01:00:11,760 So let me go through similar kinds of examples, 913 01:00:11,760 --> 01:00:16,480 as I do in the notes. 914 01:00:16,480 --> 01:00:20,430 But let me maybe build up from the simplest one first, 915 01:00:20,430 --> 01:00:27,760 parity-check representation of our favorite 8, 4, 4 code. 916 01:00:27,760 --> 01:00:32,870 In this case, the parity-check matrix is the same, we can 917 01:00:32,870 --> 01:00:36,650 take it to be the same as the generator matrix. 918 01:00:36,650 --> 01:00:40,730 Since, by now, we like trellis-oriented generator 919 01:00:40,730 --> 01:00:42,590 matrices, we'll take that one. 920 01:00:48,900 --> 01:00:55,730 So to get very explicit, so the code is the set of all y, 921 01:00:55,730 --> 01:01:01,750 such that y H transpose equals 0, and y is 8-tuple. 922 01:01:08,510 --> 01:01:11,500 So it's the set of all code words that satisfy these 923 01:01:11,500 --> 01:01:12,750 parity-check constraints. 924 01:01:20,015 --> 01:01:26,840 The first one is y1 plus y2 plus y3 plus y4 equals 0. 925 01:01:26,840 --> 01:01:31,670 That's orthogonality to this first code word. 926 01:01:31,670 --> 01:01:40,760 The second one is y2 plus y4 plus y5 plus y7 equals 0. 927 01:01:40,760 --> 01:01:48,940 The third equation is y3 plus y4 plus y5 plus y6 equals 0. 928 01:01:48,940 --> 01:01:59,310 And the fourth is y5 plus y6 plus y7 plus y8 equals 0. 929 01:01:59,310 --> 01:02:02,920 If I have any 8-tuple y, such that the elements of the 930 01:02:02,920 --> 01:02:06,000 8-tuples satisfy these four equations, then it's a code 931 01:02:06,000 --> 01:02:08,420 word, if and only if. 932 01:02:08,420 --> 01:02:10,740 A very explicit description of the code. 933 01:02:14,550 --> 01:02:22,895 Now let's go onto a graph of behavioral realization. 934 01:02:30,760 --> 01:02:37,720 So it's very natural to draw the following kind of graph of 935 01:02:37,720 --> 01:02:39,680 this realization. 936 01:02:39,680 --> 01:02:40,930 We let the y's -- 937 01:02:44,120 --> 01:02:48,160 y1, y2, y3, y4 -- 938 01:02:48,160 --> 01:02:51,360 sorry, I've changed the index set, haven't I? 939 01:02:51,360 --> 01:02:53,920 I get 1 through 8 now. 940 01:02:53,920 --> 01:03:00,910 I'm just going to make these into vertices of my graph. 941 01:03:05,020 --> 01:03:11,480 So here we have symbols, and I'm going to make a bipartite 942 01:03:11,480 --> 01:03:16,190 graph, and over here I'm going to have equations, or 943 01:03:16,190 --> 01:03:21,140 constraints, or checks. 944 01:03:21,140 --> 01:03:27,340 And I have four checks, which I'm going to draw like this. 945 01:03:33,490 --> 01:03:38,440 The first check is that y1 plus y2 plus y3 plus y4 equals 946 01:03:38,440 --> 01:03:40,490 0, so I'll just draw that like this. 947 01:03:43,800 --> 01:03:45,660 That means the same thing as this. 948 01:03:50,460 --> 01:03:58,520 The second one is y2 plus y4 plus y5 plus y7 equals 0. 949 01:03:58,520 --> 01:04:02,580 And the third one is these four -- 950 01:04:02,580 --> 01:04:06,420 two, three, four. 951 01:04:06,420 --> 01:04:08,883 And the last one is these four. 952 01:04:08,883 --> 01:04:13,230 This, this, this, this. 953 01:04:13,230 --> 01:04:15,960 It's done more nicely in the notes. 954 01:04:15,960 --> 01:04:24,130 This is called a Tanner graph, after an extremely good paper 955 01:04:24,130 --> 01:04:31,410 by Michael Tanner in 1981, which was about the only paper 956 01:04:31,410 --> 01:04:34,300 of any consequence in this subject between Gallager's 957 01:04:34,300 --> 01:04:38,120 thesis in 1961 and the rediscovery of low-density 958 01:04:38,120 --> 01:04:43,370 parity-check codes around 1995. 959 01:04:43,370 --> 01:04:46,100 That's a graphical picture of these equations. 960 01:04:46,100 --> 01:04:48,280 And what do we think of? 961 01:04:48,280 --> 01:04:51,250 You can think of testing an 8-tuple. 962 01:04:51,250 --> 01:04:56,560 Take an arbitrary, binary 8-tuple, think of these as 963 01:04:56,560 --> 01:05:00,310 memory elements, put the bits of that 8-tuple in here, and 964 01:05:00,310 --> 01:05:04,370 then ask if all these checks are satisfied. 965 01:05:04,370 --> 01:05:06,720 Since we're talking mod 2 arithmetic, we're basically 966 01:05:06,720 --> 01:05:11,500 asking if there are an even number of 1s attached to each 967 01:05:11,500 --> 01:05:14,340 of these checks. 968 01:05:14,340 --> 01:05:16,940 If and only if that's true, we've got a code 8-tuple. 969 01:05:19,520 --> 01:05:23,750 From the set of all 256 8-tuples, we find 16 that 970 01:05:23,750 --> 01:05:28,030 actually satisfy these checks, and that's the code. 971 01:05:28,030 --> 01:05:29,400 That's one way of describing the code. 972 01:05:32,440 --> 01:05:35,150 So that's how the graph is associated with local 973 01:05:35,150 --> 01:05:36,080 constraints. 974 01:05:36,080 --> 01:05:39,700 Well call them local, each of these constraints, because 975 01:05:39,700 --> 01:05:44,240 each of them only involves four of the symbols. 976 01:05:44,240 --> 01:05:48,670 In a much larger graph, a low-density parity-check code, 977 01:05:48,670 --> 01:05:51,586 the idea is that there are not very many symbols that are 978 01:05:51,586 --> 01:05:53,200 checked by each check. 979 01:05:53,200 --> 01:05:57,430 The graph is sparse, in that sense. 980 01:05:57,430 --> 01:05:59,110 But we're not quite there yet. 981 01:06:03,640 --> 01:06:07,080 So that's pretty simple. 982 01:06:07,080 --> 01:06:11,105 Let me do a generator representation. 983 01:06:11,105 --> 01:06:16,620 This is going to be a little bit more complicated because 984 01:06:16,620 --> 01:06:19,070 we have these auxiliary input variables. 985 01:06:21,830 --> 01:06:26,870 But that doesn't really complicate it very much. 986 01:06:26,870 --> 01:06:31,660 I keep using this 8, 4, 4 code just to emphasize that a 987 01:06:31,660 --> 01:06:34,600 single code can be described in many different ways. 988 01:06:34,600 --> 01:06:37,260 We're going to wind up with half a dozen different 989 01:06:37,260 --> 01:06:42,830 graphical representations of this code, and they can be 990 01:06:42,830 --> 01:06:44,946 grouped according to style. 991 01:06:50,625 --> 01:06:54,370 So again, we could use our favorite generator matrix for 992 01:06:54,370 --> 01:06:56,336 our favorite code. 993 01:06:56,336 --> 01:06:57,625 It looks like that. 994 01:07:00,660 --> 01:07:02,010 Now what do we mean? 995 01:07:02,010 --> 01:07:04,770 What are our constraints now? 996 01:07:04,770 --> 01:07:12,380 Well, now we have some hidden variables, ui, and we have y1 997 01:07:12,380 --> 01:07:14,010 is equal to u1. 998 01:07:14,010 --> 01:07:18,820 We're going to basically multiply this by u1. 999 01:07:18,820 --> 01:07:23,820 I've got some u1, u2, u3, u4 multiplying this. 1000 01:07:23,820 --> 01:07:36,600 So y1 is u1, y2 is u1 plus u2, y3 is equal to u1 plus u3, y4 1001 01:07:36,600 --> 01:07:42,430 is equal to u1 plus u2 plus u3, so forth. 1002 01:07:42,430 --> 01:07:51,930 y5 is u2 plus u3 plus u4, y6 is u2 plus u4. 1003 01:07:55,070 --> 01:07:56,710 I jumped one here. 1004 01:07:56,710 --> 01:08:03,070 This is u3 plus u4, y7 is u2 plus u4, and y8 equals u8. 1005 01:08:06,730 --> 01:08:10,350 And you will agree that the code is described, if I can 1006 01:08:10,350 --> 01:08:17,060 find any pair, (y,u), such that this is satisfied, then y 1007 01:08:17,060 --> 01:08:18,310 is an element of the code. 1008 01:08:21,540 --> 01:08:25,160 I can choose u freely, that is the input. 1009 01:08:25,160 --> 01:08:29,560 If I find a y at any u, such that these equations are 1010 01:08:29,560 --> 01:08:33,399 satisfied, then I've found a code word. 1011 01:08:33,399 --> 01:08:35,130 And that's an if and only if. 1012 01:08:35,130 --> 01:08:38,300 So that's another characterization of the code. 1013 01:08:38,300 --> 01:08:46,380 And its Tanner graph looks like this. 1014 01:08:46,380 --> 01:08:49,580 Here in this case, we have hidden variables. 1015 01:08:49,580 --> 01:08:53,060 So let's put the inputs-- 1016 01:08:53,060 --> 01:08:57,540 or, we'll see they can also be thought of as states, but 1017 01:08:57,540 --> 01:08:59,210 they're hidden variables that we don't actually 1018 01:08:59,210 --> 01:09:00,790 see in the code word. 1019 01:09:00,790 --> 01:09:05,689 u1, u2, u3, u4. 1020 01:09:05,689 --> 01:09:11,310 You can think of these as free-driving variables. 1021 01:09:11,310 --> 01:09:16,950 And to get y1, here's what we do. 1022 01:09:16,950 --> 01:09:20,920 We're going to take y1 and we're going to create a mod 2 1023 01:09:20,920 --> 01:09:23,243 sum of some of these inputs. 1024 01:09:23,243 --> 01:09:26,080 In this case, it's just u1, so I can draw it 1025 01:09:26,080 --> 01:09:27,720 as a straight through. 1026 01:09:27,720 --> 01:09:33,649 But in general, I'm going to have to make a combination to 1027 01:09:33,649 --> 01:09:35,020 get all of these. 1028 01:09:35,020 --> 01:09:38,100 So I have eight symbols over here -- 1029 01:09:38,100 --> 01:09:40,870 code symbols -- 1030 01:09:40,870 --> 01:09:46,170 and again, some constraints that have to be satisfied. 1031 01:09:46,170 --> 01:10:00,840 Equations, mod 2 sums, y8. 1032 01:10:00,840 --> 01:10:04,650 And I just, again, draw a picture. 1033 01:10:04,650 --> 01:10:10,380 y2 is u1 plus u2, so u1 plus u2. 1034 01:10:10,380 --> 01:10:13,260 these sum to give y2. 1035 01:10:13,260 --> 01:10:17,560 And so, again, if I put down some u's, put down some y's, 1036 01:10:17,560 --> 01:10:23,320 test whether all these sums are correct, then I will have 1037 01:10:23,320 --> 01:10:29,150 tested whether I found a valid behavior or trajectory. 1038 01:10:29,150 --> 01:10:33,010 Next one is u1 plus u3. 1039 01:10:33,010 --> 01:10:38,820 The next one is u1, u2, u3. 1040 01:10:38,820 --> 01:10:43,580 The next one is u2, u3, u4. 1041 01:10:43,580 --> 01:10:47,960 The next one is u3, u4. 1042 01:10:47,960 --> 01:10:51,440 The next one is u2, u4. 1043 01:10:51,440 --> 01:10:53,178 And the last one is just that one. 1044 01:10:57,710 --> 01:11:01,750 OK, so that's another graph whose constraints 1045 01:11:01,750 --> 01:11:03,150 describe the code word. 1046 01:11:06,060 --> 01:11:08,740 Here it's helpful to start to introduce the following 1047 01:11:08,740 --> 01:11:09,360 convention. 1048 01:11:09,360 --> 01:11:12,840 We fill in the circle of the variables that we actually 1049 01:11:12,840 --> 01:11:16,690 want to see, and we leave these open, the free 1050 01:11:16,690 --> 01:11:17,940 variables over here. 1051 01:11:21,970 --> 01:11:24,440 Some people would call it a generalized Tanner graph. 1052 01:11:24,440 --> 01:11:29,290 In the notes, I just call this Tanner graph 2. 1053 01:11:29,290 --> 01:11:32,290 Tanner didn't actually consider state variables. 1054 01:11:36,570 --> 01:11:40,376 This was introduced into graphical models by a guy 1055 01:11:40,376 --> 01:11:45,990 named Wiberg in his PhD thesis in Sweden in about 1995. 1056 01:11:45,990 --> 01:11:48,320 And that was a huge contribution, to have hidden 1057 01:11:48,320 --> 01:11:51,360 variables as well as observed variables, because otherwise 1058 01:11:51,360 --> 01:11:54,150 you couldn't really draw a generator representation as a 1059 01:11:54,150 --> 01:11:57,640 graph, or a trellis representation. 1060 01:11:57,640 --> 01:11:58,910 You need hidden state variables. 1061 01:12:01,720 --> 01:12:07,810 So those are two graphical styles, and the idea here is 1062 01:12:07,810 --> 01:12:13,860 that every set of 12 variables, such that these 1063 01:12:13,860 --> 01:12:17,090 eight equations are satisfied, gives you a 1064 01:12:17,090 --> 01:12:18,340 legitimate code word. 1065 01:12:21,040 --> 01:12:25,650 Here you can easily think of this as -- 1066 01:12:25,650 --> 01:12:28,650 you notice that we've described these as passive 1067 01:12:28,650 --> 01:12:29,900 constraints. 1068 01:12:31,720 --> 01:12:35,020 It's not an input-output, block diagram type of 1069 01:12:35,020 --> 01:12:36,680 representation. 1070 01:12:36,680 --> 01:12:40,490 It causes cause-effects. 1071 01:12:40,490 --> 01:12:43,310 In this case, it implicitly is. 1072 01:12:43,310 --> 01:12:46,940 How would you actually implement this generator 1073 01:12:46,940 --> 01:12:49,400 representation? 1074 01:12:49,400 --> 01:12:52,980 You implement it by picking four input bits and then 1075 01:12:52,980 --> 01:12:55,230 seeing what code words they generate. 1076 01:12:55,230 --> 01:12:58,910 In other words, there's a very definite input-output relation 1077 01:12:58,910 --> 01:13:03,220 here, where we could make this into a directed graph by 1078 01:13:03,220 --> 01:13:05,900 drawing arrows here. 1079 01:13:05,900 --> 01:13:09,030 In general, in our graphical models, we're 1080 01:13:09,030 --> 01:13:10,140 not going to do this. 1081 01:13:10,140 --> 01:13:15,490 The behavioral style is very much not in this spirit. 1082 01:13:15,490 --> 01:13:18,970 But in this case, we clearly can. 1083 01:13:18,970 --> 01:13:22,690 So if you actually wanted to generate all the code words -- 1084 01:13:22,690 --> 01:13:26,440 that's why it's called a generator representation -- 1085 01:13:26,440 --> 01:13:30,430 you simply jump through all 16 possibilities here and you 1086 01:13:30,430 --> 01:13:34,300 generate all 16 code words. 1087 01:13:34,300 --> 01:13:37,430 Or if you wanted to do a simulation, generate code 1088 01:13:37,430 --> 01:13:40,630 words, this is the way you would do it. 1089 01:13:40,630 --> 01:13:44,530 This has easy cause and effect style. 1090 01:13:44,530 --> 01:13:48,530 The kernel representation is much more implicit. 1091 01:13:48,530 --> 01:13:52,930 What is the cause and the effect here? 1092 01:13:52,930 --> 01:13:56,960 In this case, it's actually possible to find a cause and 1093 01:13:56,960 --> 01:14:01,380 effect representation. 1094 01:14:01,380 --> 01:14:07,190 It turns out that these four bits can be taken as 1095 01:14:07,190 --> 01:14:10,740 information bits, or these four places form an 1096 01:14:10,740 --> 01:14:13,960 information set. 1097 01:14:13,960 --> 01:14:19,620 So suppose we choose these four bits, and it turns out 1098 01:14:19,620 --> 01:14:25,570 that for this graph, we can trace through as follows. 1099 01:14:25,570 --> 01:14:29,660 These three bits going into this zero-sum, 1100 01:14:29,660 --> 01:14:32,600 these all go in. 1101 01:14:32,600 --> 01:14:34,380 That determines this output, right? 1102 01:14:38,710 --> 01:14:42,890 d minus 1, if d is the degree of the zero-sum node, any d 1103 01:14:42,890 --> 01:14:48,750 minus 1 inputs determine the last output, because the 1104 01:14:48,750 --> 01:14:50,200 parity-check has to check. 1105 01:14:50,200 --> 01:14:54,270 So that determines this one, which then determines y4. 1106 01:14:59,070 --> 01:15:04,594 Now, knowing that, we know all but one input here, because I 1107 01:15:04,594 --> 01:15:06,540 also have this one. 1108 01:15:06,540 --> 01:15:11,180 And that determines this, and I think it probably also 1109 01:15:11,180 --> 01:15:14,370 determines this. 1110 01:15:14,370 --> 01:15:18,170 I now have three of the four inputs here, 1111 01:15:18,170 --> 01:15:20,750 so I can get this. 1112 01:15:20,750 --> 01:15:26,970 And anyway, by tracing through it I can create a directed 1113 01:15:26,970 --> 01:15:29,270 graph that gives me a cause and effect 1114 01:15:29,270 --> 01:15:31,920 relationship from here. 1115 01:15:31,920 --> 01:15:36,110 But it's much more implicit and it can't always be done. 1116 01:15:36,110 --> 01:15:40,460 Sometimes you can find an information set that works 1117 01:15:40,460 --> 01:15:42,950 with these particular set of equations and sometimes you 1118 01:15:42,950 --> 01:15:46,440 can't, sometimes you get stuck. 1119 01:15:46,440 --> 01:15:49,050 We can come back to this when we talk about binary erasure 1120 01:15:49,050 --> 01:15:50,300 channel and stopping sets. 1121 01:15:53,270 --> 01:15:56,180 In this case, given these four, you can determine the 1122 01:15:56,180 --> 01:15:58,845 remaining four. 1123 01:15:58,845 --> 01:16:05,420 If you think of it in equation terms, if you're given y1, y2, 1124 01:16:05,420 --> 01:16:08,530 y3, you can basically go through a Gaussian elimination 1125 01:16:08,530 --> 01:16:14,325 of these equations and find y4, y6, y7 and y8. 1126 01:16:17,080 --> 01:16:21,220 So you can regard y1, y2, y3, and y5 as inputs. 1127 01:16:24,170 --> 01:16:27,270 This is much more the behavioral style, where it's 1128 01:16:27,270 --> 01:16:32,800 just simply any n-tuple that satisfies the constraints is a 1129 01:16:32,800 --> 01:16:34,780 legitimate n-tuple. 1130 01:16:34,780 --> 01:16:38,100 This is more cause and effect style over here. 1131 01:16:42,360 --> 01:16:46,610 Since we're at the end, let me just introduce one more thing. 1132 01:16:51,450 --> 01:16:53,170 This is a bipartite graph. 1133 01:16:53,170 --> 01:16:56,980 We basically have a graph with two types of nodes, one 1134 01:16:56,980 --> 01:17:01,640 representing variables, one representing constraints. 1135 01:17:01,640 --> 01:17:06,980 And the edges always go between a variable and a 1136 01:17:06,980 --> 01:17:08,520 constraint. 1137 01:17:08,520 --> 01:17:12,010 So that's what we mean by a bipartite graph. 1138 01:17:12,010 --> 01:17:14,690 There are two types of vertices, and all the edges 1139 01:17:14,690 --> 01:17:20,240 connect one type of vertex to another type of vertex. 1140 01:17:20,240 --> 01:17:22,420 So is this over here. 1141 01:17:22,420 --> 01:17:26,932 I've drawn it in a way where it's not so obvious. 1142 01:17:26,932 --> 01:17:30,040 In a generalized Tanner graph, you'd put all the variables 1143 01:17:30,040 --> 01:17:31,900 over on one side and the constraints over 1144 01:17:31,900 --> 01:17:32,600 on the other side. 1145 01:17:32,600 --> 01:17:38,800 In this case, these observed variables go naturally with 1146 01:17:38,800 --> 01:17:40,950 the constraints, so I put them over here. 1147 01:17:40,950 --> 01:17:43,730 But I could have put them over here and you would see this is 1148 01:17:43,730 --> 01:17:45,600 also a bipartite graph. 1149 01:17:48,560 --> 01:17:52,870 So this is the general character. 1150 01:17:52,870 --> 01:17:55,970 What do we think of the edges as doing in this graph? 1151 01:17:55,970 --> 01:18:00,265 We think of the edges as conducting the variables that 1152 01:18:00,265 --> 01:18:02,510 they're associated with. 1153 01:18:02,510 --> 01:18:09,050 So all of these edges, in fact, convey u1 to this 1154 01:18:09,050 --> 01:18:10,490 constraint. 1155 01:18:10,490 --> 01:18:14,030 We could, if we wanted to, label all of the edges with 1156 01:18:14,030 --> 01:18:15,890 their associated variables. 1157 01:18:15,890 --> 01:18:18,530 That's what they mean. 1158 01:18:18,530 --> 01:18:24,270 They mean here that u1 equals y1, or that u1 1159 01:18:24,270 --> 01:18:27,880 plus u2 equals y2. 1160 01:18:27,880 --> 01:18:33,580 So the edges just serve for communication in this graph. 1161 01:18:33,580 --> 01:18:38,000 There's another style of graph, which I introduced, and 1162 01:18:38,000 --> 01:18:40,450 therefore, people -- 1163 01:18:40,450 --> 01:18:43,470 not me -- but other people have called it a Forney graph. 1164 01:18:43,470 --> 01:18:44,725 I call it a normal graph. 1165 01:18:48,940 --> 01:18:56,520 The difference is that basically the vertices all 1166 01:18:56,520 --> 01:19:05,365 equal constraints, and the edges represent variables. 1167 01:19:09,490 --> 01:19:14,050 And it has certain advantages that have led to it being used 1168 01:19:14,050 --> 01:19:14,650 more and more. 1169 01:19:14,650 --> 01:19:20,770 Let me just indicate how we go from a graph of this style to 1170 01:19:20,770 --> 01:19:22,610 a graph of this style. 1171 01:19:22,610 --> 01:19:26,550 We basically just make all the variables into equality 1172 01:19:26,550 --> 01:19:28,590 constraints. 1173 01:19:28,590 --> 01:19:40,210 So in this case, I draw y1, y2, and so forth. 1174 01:19:40,210 --> 01:19:43,580 I put a little dongle on them to indicate that they 1175 01:19:43,580 --> 01:19:46,450 communicate with the outside world. 1176 01:19:46,450 --> 01:19:49,000 They are observed, external variables. 1177 01:19:52,110 --> 01:19:54,840 And then I put a little equality constraint that says 1178 01:19:54,840 --> 01:20:00,100 all of the variables attached to this constraint have to be 1179 01:20:00,100 --> 01:20:03,430 the same, which is really doing the same thing. 1180 01:20:03,430 --> 01:20:07,630 Otherwise, the topology of the graph looks the same. 1181 01:20:07,630 --> 01:20:12,560 These are, again, zero-sum constraints, or parity-check 1182 01:20:12,560 --> 01:20:15,400 constraints, and the connections are 1183 01:20:15,400 --> 01:20:17,520 done just as before. 1184 01:20:33,475 --> 01:20:35,700 I forget. 1185 01:20:35,700 --> 01:20:37,750 I'm not going to do them all out. 1186 01:20:37,750 --> 01:20:42,100 The only change, in this case, is to replace external 1187 01:20:42,100 --> 01:20:44,710 variables by an equality constraint in this. 1188 01:20:44,710 --> 01:20:47,930 So the equality constraint basically says this edge still 1189 01:20:47,930 --> 01:20:52,440 equals y1, this edge still equals y2. 1190 01:20:52,440 --> 01:20:54,930 That's what the equality constraint means. 1191 01:20:54,930 --> 01:20:56,470 Everything tied to that equality 1192 01:20:56,470 --> 01:20:58,630 constraint has to be equal. 1193 01:20:58,630 --> 01:21:03,260 So we can think of these as just replicas of this original 1194 01:21:03,260 --> 01:21:06,280 variable out here. 1195 01:21:06,280 --> 01:21:10,960 So far, you don't see any reason to prefer this. 1196 01:21:10,960 --> 01:21:18,100 If I do it in this graph, then, again, I get equality 1197 01:21:18,100 --> 01:21:20,090 constraints here. 1198 01:21:20,090 --> 01:21:26,958 I get zero-sum constraints, again, over here. 1199 01:21:26,958 --> 01:21:28,425 Eight of them. 1200 01:21:31,360 --> 01:21:34,070 And in this case, I can just represent the external 1201 01:21:34,070 --> 01:21:38,390 variables directly by these little dongles. 1202 01:21:38,390 --> 01:21:40,487 And again, I have the same graph here. 1203 01:21:43,680 --> 01:21:45,950 You still probably don't see that there's very much 1204 01:21:45,950 --> 01:21:47,420 difference and, in fact, I'd agree. 1205 01:21:47,420 --> 01:21:52,620 There's hardly any difference between these two styles. 1206 01:21:52,620 --> 01:21:53,870 So forth. 1207 01:22:00,230 --> 01:22:03,030 This is the normal graph that's equivalent to this 1208 01:22:03,030 --> 01:22:06,140 Tanner graph, it's obtained in the same way. 1209 01:22:06,140 --> 01:22:09,020 If I wanted to do it more painstakingly, I would put 1210 01:22:09,020 --> 01:22:12,030 equality constraints for each of these, I'd put little 1211 01:22:12,030 --> 01:22:14,990 external variables, and then I'd see I don't really need 1212 01:22:14,990 --> 01:22:16,260 the equality constraints. 1213 01:22:16,260 --> 01:22:20,810 So I can just compress it into y1 through y8. 1214 01:22:25,590 --> 01:22:29,680 The advantages, I will say, of the normal graph will appear 1215 01:22:29,680 --> 01:22:30,690 as we go along. 1216 01:22:30,690 --> 01:22:36,280 The main advantage is that you can prove a very nice duality 1217 01:22:36,280 --> 01:22:39,340 theorem for normal graphs -- 1218 01:22:39,340 --> 01:22:44,830 that basically shows the duality between generator and 1219 01:22:44,830 --> 01:22:46,080 parity-check constraints. 1220 01:22:46,080 --> 01:22:48,750 But to say anything about that right now would just 1221 01:22:48,750 --> 01:22:54,200 hand-waving, so we'll come back to this next week. 1222 01:22:54,200 --> 01:22:56,710 See you in a week, have a nice weekend.