1 00:00:02,090 --> 00:00:06,120 PROFESSOR: We're into chapter 11. 2 00:00:06,120 --> 00:00:10,520 I've asked Ashish to hand out today a revision of the 3 00:00:10,520 --> 00:00:14,140 previous version of chapter 11, plus chapter 12 on the 4 00:00:14,140 --> 00:00:18,430 sum-product algorithm, in case we get to it. 5 00:00:18,430 --> 00:00:21,990 I'm giving you a complete copy of the problem seven solutions 6 00:00:21,990 --> 00:00:26,240 for both weeks, and to get back on track, we're handing 7 00:00:26,240 --> 00:00:28,740 out problem set eight. 8 00:00:28,740 --> 00:00:31,520 There probably will only be one more problem set 9 00:00:31,520 --> 00:00:33,830 that you hand in. 10 00:00:33,830 --> 00:00:34,260 All right. 11 00:00:34,260 --> 00:00:40,350 We've been getting to the centerpiece of the second half 12 00:00:40,350 --> 00:00:43,300 of the course which is codes on graphs, which is the way 13 00:00:43,300 --> 00:00:48,430 we're going to get to capacity-achieving codes by 14 00:00:48,430 --> 00:00:52,840 building them on graphs of linear complexity and decoding 15 00:00:52,840 --> 00:00:56,750 them on the graphs with iterative decoding. 16 00:00:56,750 --> 00:01:00,980 I frame this in the language of behavioral realizations. 17 00:01:00,980 --> 00:01:07,180 I say a graph is a realization of a code via a behavior, 18 00:01:07,180 --> 00:01:10,250 which is the set of all the possible 19 00:01:10,250 --> 00:01:12,040 trajectories on the graph. 20 00:01:12,040 --> 00:01:17,030 So this is, again, putting it in system theory language. 21 00:01:17,030 --> 00:01:20,240 I want to distinguish between two types of variables. 22 00:01:20,240 --> 00:01:23,890 The external ones, which communicate 23 00:01:23,890 --> 00:01:25,860 with the outside world-- 24 00:01:25,860 --> 00:01:29,930 in our case, these will be the symbols in a code word. 25 00:01:29,930 --> 00:01:33,350 These are the ones that are actually involved in the code. 26 00:01:33,350 --> 00:01:35,900 So these are more or less pre-specified for us when 27 00:01:35,900 --> 00:01:38,470 we're trying to realize a code. 28 00:01:38,470 --> 00:01:42,660 But we can adjoin auxiliary variables, internal variables, 29 00:01:42,660 --> 00:01:45,510 we often call them state variables, hidden variables, 30 00:01:45,510 --> 00:01:48,920 latent variables, which are really there for our 31 00:01:48,920 --> 00:01:49,730 convenience. 32 00:01:49,730 --> 00:01:52,910 They're to make the realization simpler, or more 33 00:01:52,910 --> 00:01:57,090 elegant in some way, or to have properties, or whatever. 34 00:01:57,090 --> 00:02:00,700 But these are free for us to have because they're inside 35 00:02:00,700 --> 00:02:05,190 the box and the world never sees them. 36 00:02:05,190 --> 00:02:09,840 Now, we describe the behavior by a set of constraints on the 37 00:02:09,840 --> 00:02:13,590 variables, which in our case, these are local constraints 38 00:02:13,590 --> 00:02:17,200 that only involves a few of the variables, each one. 39 00:02:17,200 --> 00:02:21,250 The idea is to express a global behavior via local 40 00:02:21,250 --> 00:02:24,000 constraints on variables. 41 00:02:24,000 --> 00:02:28,410 And in our case, the constraints will be expressed 42 00:02:28,410 --> 00:02:32,650 by equations, as we've seen, or in a little bit more 43 00:02:32,650 --> 00:02:38,000 generality by little codes, which will go in to make up 44 00:02:38,000 --> 00:02:39,310 this big code. 45 00:02:39,310 --> 00:02:41,960 So in some sense, you can think of codes on graphs as 46 00:02:41,960 --> 00:02:45,150 being making big codes out of little codes, making global 47 00:02:45,150 --> 00:02:48,890 codes out of local codes. 48 00:02:48,890 --> 00:02:52,290 The behavior, then, is simply defined as the set of all 49 00:02:52,290 --> 00:02:56,330 combinations of variables, both external and internal, 50 00:02:56,330 --> 00:02:58,700 that satisfy all the constraints. 51 00:02:58,700 --> 00:03:02,390 If we can find a set of values for all the variables such 52 00:03:02,390 --> 00:03:05,730 that all these local constraints are satisfied, 53 00:03:05,730 --> 00:03:12,430 then that's called a valid trajectory, or whatever. 54 00:03:12,430 --> 00:03:14,830 And that's the entire behavior. 55 00:03:14,830 --> 00:03:21,090 And the code is simply the set of all external variable 56 00:03:21,090 --> 00:03:24,930 n-tuples that occur as parts of valid behaviors-- 57 00:03:24,930 --> 00:03:27,370 in other words, the projection of the behavior just onto the 58 00:03:27,370 --> 00:03:28,990 external variables. 59 00:03:28,990 --> 00:03:32,940 All right, so it's an elegant and rather implicit way of 60 00:03:32,940 --> 00:03:37,750 describing the code, but we'll see it's very, very powerful. 61 00:03:37,750 --> 00:03:42,210 And last time, we talked about various styles of 62 00:03:42,210 --> 00:03:44,260 realizations. 63 00:03:44,260 --> 00:03:49,160 I gave you examples of generator representations and 64 00:03:49,160 --> 00:03:50,770 parity-check representations. 65 00:03:50,770 --> 00:03:53,390 This is really where it all started, with parity-check 66 00:03:53,390 --> 00:03:56,610 representations, where this style of graphical 67 00:03:56,610 --> 00:04:00,720 representation is very natural. 68 00:04:00,720 --> 00:04:05,950 You've got the symbols in the code, those are the only 69 00:04:05,950 --> 00:04:08,340 symbols you need, and you've got parity checks, and the 70 00:04:08,340 --> 00:04:10,360 parity checks constrain the symbols. 71 00:04:10,360 --> 00:04:13,350 If you've got n minus k parity checks, you get n minus k 72 00:04:13,350 --> 00:04:14,590 constraints. 73 00:04:14,590 --> 00:04:18,610 And it's a kernel representation in the language 74 00:04:18,610 --> 00:04:22,290 of linear algebra. 75 00:04:22,290 --> 00:04:28,120 It's simply the set of all code n-tuples that satisfy all 76 00:04:28,120 --> 00:04:29,290 the parity checks. 77 00:04:29,290 --> 00:04:32,260 That's it, we don't need any hidden variables. 78 00:04:32,260 --> 00:04:36,620 In a generator representation, this is a more natural 79 00:04:36,620 --> 00:04:38,940 representation, when we're trying to generate 80 00:04:38,940 --> 00:04:41,360 or simulate a code. 81 00:04:41,360 --> 00:04:47,530 We want to run through a set of inputs that in effect 82 00:04:47,530 --> 00:04:50,340 create all the outputs, all the code words. 83 00:04:50,340 --> 00:04:52,720 In this case, we don't actually see the inputs as 84 00:04:52,720 --> 00:04:54,590 part of the code words, necessarily. 85 00:04:54,590 --> 00:04:56,230 We might, we might not. 86 00:04:56,230 --> 00:04:59,400 But we distinguish them in this representation and we 87 00:04:59,400 --> 00:05:02,620 regard the inputs as hidden variables, or state variables, 88 00:05:02,620 --> 00:05:03,850 if you like. 89 00:05:03,850 --> 00:05:06,340 They're part of the realization, but they don't 90 00:05:06,340 --> 00:05:09,050 actually occur in the code words, so they fit into the 91 00:05:09,050 --> 00:05:11,940 hidden category. 92 00:05:11,940 --> 00:05:16,990 OK, and we talked about two styles of graphs. 93 00:05:16,990 --> 00:05:20,400 Tanner graphs, which are older and more established. 94 00:05:20,400 --> 00:05:22,870 Tanner really wrote the foundation paper in this 95 00:05:22,870 --> 00:05:25,030 subject in 1981. 96 00:05:25,030 --> 00:05:26,930 I regard him as the founder of the 97 00:05:26,930 --> 00:05:29,340 subject of codes on graphs. 98 00:05:29,340 --> 00:05:31,850 However, like a lot of good papers, his 99 00:05:31,850 --> 00:05:34,360 was completely ignored-- 100 00:05:34,360 --> 00:05:45,540 that's almost true-- for 15 years. 101 00:05:45,540 --> 00:05:49,210 And then it was rediscovered around 1995, when people began 102 00:05:49,210 --> 00:05:52,120 to wake up to this subject again. 103 00:05:52,120 --> 00:05:55,490 And Tanner graphs really come out of this idea. 104 00:05:55,490 --> 00:05:57,570 They were closely associated with that. 105 00:05:57,570 --> 00:06:00,290 We can generalize them to be more like this. 106 00:06:00,290 --> 00:06:01,960 But it's this simple idea. 107 00:06:01,960 --> 00:06:05,280 We have a bipartite graph, we make one class of vertices 108 00:06:05,280 --> 00:06:09,510 into the variables, one class into the constraints, and we 109 00:06:09,510 --> 00:06:13,170 simply tie a variable to a constraint when it's involved 110 00:06:13,170 --> 00:06:14,430 in the constraint. 111 00:06:14,430 --> 00:06:18,280 And we can generalize it by making internal and external 112 00:06:18,280 --> 00:06:20,410 variables, which we have to distinguish in the 113 00:06:20,410 --> 00:06:22,160 picture in some way. 114 00:06:22,160 --> 00:06:27,180 And the constraints, initially, they were just 115 00:06:27,180 --> 00:06:29,380 zero-sum constraints and parity-check codes. 116 00:06:29,380 --> 00:06:31,290 One of the things Tanner did was he said, well, they could 117 00:06:31,290 --> 00:06:32,950 be any codes. 118 00:06:32,950 --> 00:06:37,970 A single parity check is like a single parity-check code. 119 00:06:37,970 --> 00:06:41,620 And we can make any codes the constraints. 120 00:06:41,620 --> 00:06:44,870 And the edges don't work very hard in the Tanner graph. 121 00:06:44,870 --> 00:06:49,120 They implicitly carry the variables to the constraints 122 00:06:49,120 --> 00:06:51,506 so we could label them with the variables. 123 00:06:51,506 --> 00:06:54,610 And we showed how a Tanner graph could always be 124 00:06:54,610 --> 00:06:58,480 transformed to this normal graph, which I think is 125 00:06:58,480 --> 00:07:01,890 certainly in more complicated situations, a cleaner way of 126 00:07:01,890 --> 00:07:06,480 explaining things, and cleaner in a variety of senses. 127 00:07:06,480 --> 00:07:09,780 Here is our taxonomy. 128 00:07:09,780 --> 00:07:14,370 Vertices indicate constraints, half edges 129 00:07:14,370 --> 00:07:17,180 indicate external variables. 130 00:07:17,180 --> 00:07:19,810 A half edge is something you can tie to another half edge 131 00:07:19,810 --> 00:07:22,190 to form an I/O pad. 132 00:07:22,190 --> 00:07:25,170 Edges tie together two constraints. 133 00:07:25,170 --> 00:07:27,210 They're internal, and so these represent 134 00:07:27,210 --> 00:07:29,070 the internal variables. 135 00:07:29,070 --> 00:07:31,760 And this is cleaner both analytically-- 136 00:07:31,760 --> 00:07:34,520 there's this nice duality theorem about normal graphs 137 00:07:34,520 --> 00:07:36,880 that I won't be presenting to you, and a 138 00:07:36,880 --> 00:07:38,720 couple other things. 139 00:07:38,720 --> 00:07:41,190 They're cleaner when you come to implement 140 00:07:41,190 --> 00:07:43,670 the sum-product algorithm. 141 00:07:43,670 --> 00:07:47,730 I think they're just generally cleaner, but I may be biased. 142 00:07:47,730 --> 00:07:50,490 I want you to see both because these still predominate the 143 00:07:50,490 --> 00:07:53,550 literature, although more and more people seem to be using 144 00:07:53,550 --> 00:07:55,040 this style of graph, and that's what I 145 00:07:55,040 --> 00:07:56,433 will use in the course. 146 00:07:56,433 --> 00:07:57,420 All right? 147 00:07:57,420 --> 00:08:01,820 So that's a review of where we are. 148 00:08:01,820 --> 00:08:03,710 OK. 149 00:08:03,710 --> 00:08:08,420 Today we're going to go on to other important styles of 150 00:08:08,420 --> 00:08:14,780 realizations, and particularly the next one we'll talk about 151 00:08:14,780 --> 00:08:17,490 is trellis realizations. 152 00:08:17,490 --> 00:08:22,550 So this will tie back to all the work we did in chapter 10. 153 00:08:22,550 --> 00:08:26,020 And what did we discover in chapter 10? 154 00:08:26,020 --> 00:08:29,330 The key to a trellis realization was a 155 00:08:29,330 --> 00:08:32,780 trellis-oriented generator matrix. 156 00:08:32,780 --> 00:08:40,929 And again, I will put up our favorite matrix on the board, 157 00:08:40,929 --> 00:08:44,555 the generator of the 8, 4, 4 Reed-Muller code. 158 00:08:48,440 --> 00:08:52,440 And in the form we've come to know and love, 159 00:08:52,440 --> 00:08:54,510 it looks like this. 160 00:08:54,510 --> 00:08:58,810 And what makes it trellis-oriented is that all 161 00:08:58,810 --> 00:09:00,940 the starting times are different and all the ending 162 00:09:00,940 --> 00:09:02,170 times are different. 163 00:09:02,170 --> 00:09:05,460 Therefore these generators are as short as possible. 164 00:09:05,460 --> 00:09:08,200 The main feature of this generator that we want to look 165 00:09:08,200 --> 00:09:13,580 at is the spans, the active parts of the generators, which 166 00:09:13,580 --> 00:09:15,300 we've made as short as possible. 167 00:09:15,300 --> 00:09:20,130 And let me explicitly write out what they are. 168 00:09:20,130 --> 00:09:24,330 The first generator is active from symbol time 0 169 00:09:24,330 --> 00:09:26,080 to symbol time 3. 170 00:09:26,080 --> 00:09:30,870 This one is active from 1 to 6. 171 00:09:30,870 --> 00:09:35,930 This one is active from 2 to 5. 172 00:09:35,930 --> 00:09:40,570 And this one is active from 4 to 7. 173 00:09:40,570 --> 00:09:42,380 OK, so those are the active spans. 174 00:09:42,380 --> 00:09:47,250 And we know from that, we can find the dimensions of minimal 175 00:09:47,250 --> 00:09:50,790 state spaces and of minimal branch spaces. 176 00:09:50,790 --> 00:09:53,490 OK, we just count the number of active generators at either 177 00:09:53,490 --> 00:09:56,690 of the state times, which are between the symbols, or the 178 00:09:56,690 --> 00:10:00,310 symbol times themselves for the branches. 179 00:10:00,310 --> 00:10:04,220 OK, again, quick review, especially because we have a 180 00:10:04,220 --> 00:10:07,050 few guests here who we haven't seen previously in the course. 181 00:10:07,050 --> 00:10:09,620 So that's where we've been. 182 00:10:09,620 --> 00:10:12,020 All right. 183 00:10:12,020 --> 00:10:17,730 Once you have any generator matrix, or in particular a 184 00:10:17,730 --> 00:10:21,130 trellis-oriented generator matrix, the code is simply 185 00:10:21,130 --> 00:10:27,250 described as the set of all linear combinations of the 186 00:10:27,250 --> 00:10:29,176 matrix elements. 187 00:10:29,176 --> 00:10:32,710 And in particular, the code symbol at every time is a 188 00:10:32,710 --> 00:10:36,980 linear combination of the matrix elements in a 189 00:10:36,980 --> 00:10:38,890 particular column. 190 00:10:38,890 --> 00:10:42,350 All right, so let's use that to get a trellis realization. 191 00:10:45,890 --> 00:10:47,870 I've done this in various ways. 192 00:10:47,870 --> 00:10:52,940 I'm still not satisfied, though the way that I do it in 193 00:10:52,940 --> 00:10:58,830 the notes is quite notation heavy and mathematical. 194 00:10:58,830 --> 00:11:03,240 I hope this picture of what we're going to do is going to 195 00:11:03,240 --> 00:11:08,570 give you a totally transparent idea of what's going on. 196 00:11:08,570 --> 00:11:15,770 OK, we're going to have a trellis realization. 197 00:11:15,770 --> 00:11:18,160 Let's write down the symbol times. 198 00:11:18,160 --> 00:11:21,170 And I'll write them all the way across the board. 199 00:11:21,170 --> 00:11:26,765 0, 1, 2, 3, 4, 5, 6, 7. 200 00:11:29,720 --> 00:11:39,510 And let me save the coefficients, ui, here. 201 00:11:39,510 --> 00:11:43,060 And I'll write them as equality constraints, but you 202 00:11:43,060 --> 00:11:48,200 can also think of them as a little binary shift register. 203 00:11:48,200 --> 00:11:52,500 I'm going to need u1 at four different times. 204 00:11:52,500 --> 00:11:55,060 I'm going to need u1 during its span at time 205 00:11:55,060 --> 00:11:57,410 0, 1, 2, and 3. 206 00:11:57,410 --> 00:12:01,460 So I'll set up a set of constraints here to propagate 207 00:12:01,460 --> 00:12:04,262 u1 between these three times. 208 00:12:04,262 --> 00:12:07,470 And u1 is going to be an internal variable. 209 00:12:07,470 --> 00:12:10,030 Strictly, there are three replicas of this internal 210 00:12:10,030 --> 00:12:11,930 variable here. 211 00:12:11,930 --> 00:12:14,080 And I can pull it out at each time, too. 212 00:12:14,080 --> 00:12:21,650 So this is where I'm going to keep u1, in effect. 213 00:12:21,650 --> 00:12:28,540 Similarly, u2 I'm going to keep around for all the times 214 00:12:28,540 --> 00:12:31,680 that I need it, which is these six times. 215 00:12:39,978 --> 00:12:43,590 OK, so is that six? 216 00:12:43,590 --> 00:12:45,280 That's only five. 217 00:12:45,280 --> 00:12:51,120 These are all the times that u2 is actually involved in the 218 00:12:51,120 --> 00:12:54,950 output symbols, so I'm going to have it when I need it. 219 00:12:54,950 --> 00:12:57,136 u3 I need for four times. 220 00:13:06,840 --> 00:13:10,020 I've been inconsistent in whether I write the variables 221 00:13:10,020 --> 00:13:12,730 or the values of the variables. 222 00:13:12,730 --> 00:13:14,660 In this case, I'm writing values. 223 00:13:14,660 --> 00:13:18,770 And finally, u4. 224 00:13:18,770 --> 00:13:21,250 I see I'm going to need more than one board here. 225 00:13:32,300 --> 00:13:34,580 So now I've got everything I need at the 226 00:13:34,580 --> 00:13:39,390 time that I need it. 227 00:13:39,390 --> 00:13:41,795 Now, this is going to be tricky. 228 00:13:45,260 --> 00:13:47,800 I'm simply going to have a linear combination-- 229 00:13:47,800 --> 00:13:53,710 call it g0, g1, and so forth-- 230 00:13:53,710 --> 00:14:00,810 which is going to create my output at time 0, or time 1, 231 00:14:00,810 --> 00:14:02,640 or time 2, and so forth. 232 00:14:07,170 --> 00:14:11,530 And you'll agree from this generator matrix that the 233 00:14:11,530 --> 00:14:15,870 output at time 1 is some linear combination of u1. 234 00:14:19,520 --> 00:14:24,390 And the output at time 2 is some linear combination of 235 00:14:24,390 --> 00:14:25,640 these two guys. 236 00:14:29,660 --> 00:14:36,100 OK, and the output at time 3 is some linear combination of 237 00:14:36,100 --> 00:14:42,610 these three guys, and so forth. 238 00:14:42,610 --> 00:14:44,240 You see what I'm doing? 239 00:14:44,240 --> 00:14:48,000 So I could describe these as simply a linear constraint. 240 00:14:48,000 --> 00:14:50,310 What are the possible combinations of inputs and 241 00:14:50,310 --> 00:14:52,810 outputs in this constraint? 242 00:14:52,810 --> 00:14:54,060 And similarly-- 243 00:14:59,060 --> 00:15:01,620 and don't make me draw the lines in. 244 00:15:23,100 --> 00:15:33,310 OK, so I claim this is a realization directly from this 245 00:15:33,310 --> 00:15:35,280 trellis-oriented generator matrix. 246 00:15:35,280 --> 00:15:35,820 Yes? 247 00:15:35,820 --> 00:15:42,885 AUDIENCE: In this example in the generator, g has a 0 248 00:15:42,885 --> 00:15:44,070 [UNINTELLIGIBLE]. 249 00:15:44,070 --> 00:15:44,950 PROFESSOR: Right. 250 00:15:44,950 --> 00:15:48,450 AUDIENCE: So before, the component had to be 0 in the 251 00:15:48,450 --> 00:15:49,350 linear combination? 252 00:15:49,350 --> 00:15:50,130 Or are you-- 253 00:15:50,130 --> 00:15:53,710 PROFESSOR: I can do it that way. 254 00:15:53,710 --> 00:15:57,670 Clearly, I don't need this input at this time. 255 00:15:57,670 --> 00:15:59,360 So I could erase that. 256 00:15:59,360 --> 00:16:01,940 Would that make you feel better? 257 00:16:01,940 --> 00:16:05,840 Actually, that gives an even better picture. 258 00:16:05,840 --> 00:16:09,420 It's not going to make any fundamental difference, but 259 00:16:09,420 --> 00:16:12,660 I'm doing this for a generic set of generators that cover 260 00:16:12,660 --> 00:16:14,060 these spans. 261 00:16:14,060 --> 00:16:14,430 All right? 262 00:16:14,430 --> 00:16:17,490 For the particular ones we have here, yes, I can make 263 00:16:17,490 --> 00:16:22,780 that further change and eliminate those two inputs. 264 00:16:22,780 --> 00:16:29,170 But I'm shortly going to aggregate this and put it in a 265 00:16:29,170 --> 00:16:30,600 more aggregated form. 266 00:16:30,600 --> 00:16:33,490 And what's happening internally here then won't 267 00:16:33,490 --> 00:16:36,900 matter to us very much. 268 00:16:36,900 --> 00:16:38,270 OK. 269 00:16:38,270 --> 00:16:44,970 And notice that I can regard this as the branch 270 00:16:44,970 --> 00:16:47,480 space at this time. 271 00:16:47,480 --> 00:16:51,270 And it has the right dimension for the branch space at each 272 00:16:51,270 --> 00:16:52,610 of these times. 273 00:16:52,610 --> 00:16:56,680 And let me take it a little more slowly. 274 00:16:56,680 --> 00:16:58,920 First, I claim it's a realization. 275 00:16:58,920 --> 00:17:01,730 If I built that thing with the appropriate linear 276 00:17:01,730 --> 00:17:06,640 combinations here, then the behavior of this is basically 277 00:17:06,640 --> 00:17:10,589 the set of all u's and y's that satisfy all these 278 00:17:10,589 --> 00:17:11,390 constraints. 279 00:17:11,390 --> 00:17:14,040 And if I just look at the outputs, that's the set of all 280 00:17:14,040 --> 00:17:15,950 code words. 281 00:17:15,950 --> 00:17:20,630 So I've constructed a behavioral realization in a 282 00:17:20,630 --> 00:17:22,700 certain way. 283 00:17:22,700 --> 00:17:26,579 OK, and now I'm going to start to aggregate. 284 00:17:26,579 --> 00:17:32,016 Aggregate just means drawing lines around sub-graphs. 285 00:17:32,016 --> 00:17:34,530 Or sometimes in the notes, I call this agglomeration. 286 00:17:38,190 --> 00:17:43,700 So I'm going to regard all this as one constraint on the 287 00:17:43,700 --> 00:17:47,380 variables that come out of this. 288 00:17:47,380 --> 00:17:50,140 The constraints that are affected by this 289 00:17:50,140 --> 00:17:51,020 agglomeration-- 290 00:17:51,020 --> 00:17:55,210 the variables that are affected are y0 and u1. 291 00:17:55,210 --> 00:17:58,820 OK, so this I can draw-- 292 00:17:58,820 --> 00:18:02,020 let me draw it this way-- 293 00:18:02,020 --> 00:18:06,590 as a little constraint code which operates at time 0. 294 00:18:06,590 --> 00:18:11,450 And it's a linear 2, 1 code that basically ties together 295 00:18:11,450 --> 00:18:18,910 y0 and u1, whatever the relationship between them is. 296 00:18:18,910 --> 00:18:20,950 It probably is the y0 equals u1. 297 00:18:24,400 --> 00:18:26,200 The first generator starts at this time. 298 00:18:26,200 --> 00:18:28,210 That's about the only thing it could be. 299 00:18:28,210 --> 00:18:31,990 Similarly, I'm going to aggregate all this part of the 300 00:18:31,990 --> 00:18:36,140 realization here and consider it a big block, or a big 301 00:18:36,140 --> 00:18:37,390 constraint. 302 00:18:39,160 --> 00:18:43,340 At time 1, there's some constraint that affects 303 00:18:43,340 --> 00:18:50,760 y1 and u1 and u2. 304 00:18:50,760 --> 00:18:53,990 So this is a little 4, 2 code on the 305 00:18:53,990 --> 00:18:55,550 four incident variables. 306 00:18:55,550 --> 00:18:58,640 One of the constraints is that u1 on this side 307 00:18:58,640 --> 00:19:00,520 equals u1 on this side. 308 00:19:00,520 --> 00:19:02,750 These are really two replicas of the same thing. 309 00:19:02,750 --> 00:19:04,240 So there's an equality constraint 310 00:19:04,240 --> 00:19:05,830 propagating through there. 311 00:19:05,830 --> 00:19:10,560 Then I have some function of y1 as a function of u1, which 312 00:19:10,560 --> 00:19:13,900 I'm going to regard as the state at this time. 313 00:19:13,900 --> 00:19:19,300 And u1 and u2, let me call that state space at time 1. 314 00:19:19,300 --> 00:19:22,660 Let me call this at time 0, let me call this the state 315 00:19:22,660 --> 00:19:24,630 space at time 1. 316 00:19:24,630 --> 00:19:28,810 And I did go through a development where I showed 317 00:19:28,810 --> 00:19:31,580 that these u's could be regarded as the components of 318 00:19:31,580 --> 00:19:32,570 the state spaces. 319 00:19:32,570 --> 00:19:37,730 This is a more constructive way of seeing the same thing. 320 00:19:37,730 --> 00:19:41,680 And notice they have the right dimensions. 321 00:19:41,680 --> 00:19:45,330 So the states are what carry the information from time 0 to 322 00:19:45,330 --> 00:19:47,640 time 1, just as we would want. 323 00:19:47,640 --> 00:19:50,220 They kind of embody the Markov property. 324 00:19:50,220 --> 00:19:50,662 Yes? 325 00:19:50,662 --> 00:19:53,320 AUDIENCE: Why is c0 length 2? 326 00:19:53,320 --> 00:19:56,308 PROFESSOR: Why is c0-- 327 00:19:56,308 --> 00:19:57,590 AUDIENCE: Length 2. 328 00:19:57,590 --> 00:19:58,270 PROFESSOR: Length 2. 329 00:19:58,270 --> 00:19:59,880 Because it affects two bits. 330 00:19:59,880 --> 00:20:00,940 AUDIENCE: Thank you. 331 00:20:00,940 --> 00:20:02,610 PROFESSOR: It affects these two bits. 332 00:20:06,120 --> 00:20:10,340 Let's see, how did I know that there's only one possible 333 00:20:10,340 --> 00:20:11,280 constraint here? 334 00:20:11,280 --> 00:20:14,360 Because that's the dimension of the branch space. 335 00:20:14,360 --> 00:20:18,390 It's really all determined by u1. 336 00:20:18,390 --> 00:20:21,790 Over here, however, it's determined by u1 and u2. 337 00:20:21,790 --> 00:20:26,550 So the dimension is 2. u1 and u2 are free, And the others 338 00:20:26,550 --> 00:20:29,490 are fixed once we know those. 339 00:20:29,490 --> 00:20:32,560 So I was a little ahead of myself there. 340 00:20:32,560 --> 00:20:37,180 And similarly, let's keep drawing in this fashion, 341 00:20:37,180 --> 00:20:43,660 here's a constraint code at time 2 which relates these 342 00:20:43,660 --> 00:20:48,840 possible variables, u1, u2, u3, which I'm going to call 343 00:20:48,840 --> 00:20:54,710 the state space at time 2. 344 00:20:54,710 --> 00:21:00,710 So my times are not what I expect. 345 00:21:00,710 --> 00:21:05,340 But it's always really a problem to keep the indexes 346 00:21:05,340 --> 00:21:06,440 consistent. 347 00:21:06,440 --> 00:21:14,330 We want state time 1 to occur after the first symbol. 348 00:21:14,330 --> 00:21:19,160 So this should be state time 2, this should be state time 3 349 00:21:19,160 --> 00:21:21,110 to be consistent with what's in the notes. 350 00:21:21,110 --> 00:21:22,360 I'm sorry. 351 00:21:24,740 --> 00:21:26,460 And so forth. 352 00:21:26,460 --> 00:21:28,770 What is the length of this code? 353 00:21:28,770 --> 00:21:33,960 It's simply the number of bits that it controls, which is 6. 354 00:21:33,960 --> 00:21:36,400 1, 2, 3, 4, 5, 6. 355 00:21:36,400 --> 00:21:39,290 What's the dimension? 356 00:21:39,290 --> 00:21:46,710 It's 3, because u1, u2, u3 are free if I only look at this 357 00:21:46,710 --> 00:21:48,850 constraint independent of everything else. 358 00:21:52,056 --> 00:21:55,840 OK, and continuing this way, at this point, you notice I 359 00:21:55,840 --> 00:21:59,320 only have u2 and u3. 360 00:21:59,320 --> 00:22:02,880 So my state space at time 4 has gone down in dimension. 361 00:22:05,620 --> 00:22:12,630 And here I have c3, which is, again, a 6, 3 code, and so 362 00:22:12,630 --> 00:22:13,970 forth, as I go ahead. 363 00:22:13,970 --> 00:22:18,900 Let me just draw it. 364 00:22:18,900 --> 00:22:20,995 This is also a 6, 3. 365 00:22:24,060 --> 00:22:27,635 This is time 5. 366 00:22:33,288 --> 00:22:37,400 I have the state space at time 6. 367 00:22:42,740 --> 00:22:49,250 This is back down to 1, 4. 368 00:22:49,250 --> 00:22:52,610 It's symmetrical on this side, as you know. 369 00:23:07,180 --> 00:23:10,400 So that's my trellis realization. 370 00:23:10,400 --> 00:23:14,780 And I claim this, too, is a realization, in the sense that 371 00:23:14,780 --> 00:23:19,050 any combination of u's and y's that satisfy all of these 372 00:23:19,050 --> 00:23:23,300 constraints is a legitimate trajectory. 373 00:23:23,300 --> 00:23:27,100 And the set of all y's that are part of those legitimate 374 00:23:27,100 --> 00:23:29,350 trajectories form the code. 375 00:23:29,350 --> 00:23:34,630 In fact, it's pretty explicit here that the dimension of the 376 00:23:34,630 --> 00:23:39,330 code is 4, corresponding to u1, u2, u3. 377 00:23:39,330 --> 00:23:45,040 So I have 8 outputs, I have 16 possible behaviors. 378 00:23:45,040 --> 00:23:49,270 I pick off the combinations of u's and y's that could make 379 00:23:49,270 --> 00:23:53,430 that, which is basically determined by this matrix. 380 00:23:53,430 --> 00:23:57,900 And I just pick off the y's and that's my 8, 4 code. 381 00:23:57,900 --> 00:24:03,720 OK, so that's what trellis looks like. 382 00:24:03,720 --> 00:24:06,645 The constraint code really embodies the branches. 383 00:24:09,760 --> 00:24:13,010 This encapsulates everything that can happen at time 0. 384 00:24:13,010 --> 00:24:17,370 This encapsulates everything that could happen at time 1. 385 00:24:17,370 --> 00:24:21,670 The u's are really my state spaces. 386 00:24:21,670 --> 00:24:27,180 This constrains state, output, next state, in nice linear 387 00:24:27,180 --> 00:24:28,920 system theories style. 388 00:24:28,920 --> 00:24:31,720 This tells me what combinations of state, output, 389 00:24:31,720 --> 00:24:35,020 next state, I can have at time 2, and so 390 00:24:35,020 --> 00:24:36,800 forth across the board. 391 00:24:36,800 --> 00:24:40,240 So the constraints are local constraints-- 392 00:24:40,240 --> 00:24:41,900 local in time, in this sense-- 393 00:24:41,900 --> 00:24:46,470 that constrain what can happen as you get to the next state. 394 00:24:46,470 --> 00:24:48,380 Once you get to the next state, this state has the 395 00:24:48,380 --> 00:24:49,340 Markov property. 396 00:24:49,340 --> 00:24:52,470 This is all that the memory you need of the past in order 397 00:24:52,470 --> 00:24:55,700 to determine the future, or the whole set of future 398 00:24:55,700 --> 00:24:58,380 possibilities. 399 00:24:58,380 --> 00:25:03,610 Each of these is a linear vector space 400 00:25:03,610 --> 00:25:04,660 over the ground field. 401 00:25:04,660 --> 00:25:07,620 In fact, it's just the space of 1-tuples, 2-tuples, 402 00:25:07,620 --> 00:25:10,570 3-tuples, 2-tuples, and so forth. 403 00:25:10,570 --> 00:25:14,220 And it has certain dimension, in this case, 404 00:25:14,220 --> 00:25:16,170 equal to its size. 405 00:25:16,170 --> 00:25:19,470 And if you calculate the dimensions of-- 406 00:25:19,470 --> 00:25:24,290 call this the state space, call this the branch 407 00:25:24,290 --> 00:25:27,890 constraint code if you like, or the branch space, it's 408 00:25:27,890 --> 00:25:33,190 isomorphic, the dimensions are minimal here by construction 409 00:25:33,190 --> 00:25:36,105 from the trellis-oriented generator matrix. 410 00:25:36,105 --> 00:25:41,600 If we want to know the minimal size of the state space at 411 00:25:41,600 --> 00:25:48,740 time 4 here in the center, we just calculate the number of 412 00:25:48,740 --> 00:25:52,070 generators that are active at time 4, and that's precisely 413 00:25:52,070 --> 00:25:53,850 what we're going to get over there, too, by our 414 00:25:53,850 --> 00:25:54,880 construction. 415 00:25:54,880 --> 00:25:58,045 If we want to compute the dimension of the branch space 416 00:25:58,045 --> 00:26:01,630 at time 3, it's the number of active generators 417 00:26:01,630 --> 00:26:02,890 at symbol time 3. 418 00:26:02,890 --> 00:26:06,960 There are 3 active ones, and we've just forced this to have 419 00:26:06,960 --> 00:26:08,230 that structure. 420 00:26:08,230 --> 00:26:17,350 So this is a normal graph of a minimal trellis realization of 421 00:26:17,350 --> 00:26:19,540 the 8, 4 code. 422 00:26:19,540 --> 00:26:21,410 OK? 423 00:26:21,410 --> 00:26:24,820 So we can clearly do that for any code, right? 424 00:26:24,820 --> 00:26:25,840 For any code, we can find a 425 00:26:25,840 --> 00:26:27,340 trellis-oriented generator matrix. 426 00:26:27,340 --> 00:26:30,130 We can go through these steps, and we'll always come up with 427 00:26:30,130 --> 00:26:32,061 something that looks like this down here. 428 00:26:34,770 --> 00:26:37,990 OK, now, what are the properties of this graph? 429 00:26:37,990 --> 00:26:39,240 Does it have cycles? 430 00:26:48,600 --> 00:26:50,740 Does this graph have cycles? 431 00:26:50,740 --> 00:26:52,220 Anybody? 432 00:26:52,220 --> 00:26:54,580 "No," is one answer I get. 433 00:26:54,580 --> 00:26:57,190 What's the other possible answer? 434 00:26:57,190 --> 00:27:00,200 How many people say no, and how many people say yes? 435 00:27:00,200 --> 00:27:01,400 I want to see a show of hands. 436 00:27:01,400 --> 00:27:04,275 How many people think this graph is cycle free? 437 00:27:07,160 --> 00:27:12,430 OK, and how many people think that it has cycles? 438 00:27:12,430 --> 00:27:15,860 OK, the majority is wrong. 439 00:27:15,860 --> 00:27:17,820 This graph clearly has cycles. 440 00:27:17,820 --> 00:27:19,460 Here are two edges here, and here's a cycle. 441 00:27:24,180 --> 00:27:28,380 OK, so as a graph, it has cycles because it has multiple 442 00:27:28,380 --> 00:27:30,525 edges going between these constraints. 443 00:27:33,130 --> 00:27:39,080 All right, so a good thing to do is to make this a 444 00:27:39,080 --> 00:27:45,860 cycle-free graph, and that's just a matter of regarding 445 00:27:45,860 --> 00:27:50,190 this as instead of two binary variables, we regard it as one 446 00:27:50,190 --> 00:27:53,150 quaternary variable. 447 00:27:53,150 --> 00:28:02,540 All right, so we're going to regard this as a single binary 448 00:28:02,540 --> 00:28:04,350 variable of dimension one. 449 00:28:04,350 --> 00:28:06,680 This is a quaternary state space. 450 00:28:06,680 --> 00:28:08,910 There are 4 possible states here. 451 00:28:08,910 --> 00:28:12,620 So the dimension of the state space at this time is 2. 452 00:28:12,620 --> 00:28:21,050 So when I draw abbreviations like this, you can think of 453 00:28:21,050 --> 00:28:25,550 them as being decomposable into binary 454 00:28:25,550 --> 00:28:27,660 variables, if you like. 455 00:28:27,660 --> 00:28:32,520 But the advantage of drawing them as larger variables, 456 00:28:32,520 --> 00:28:38,460 higher-valued variables, is that now does this 457 00:28:38,460 --> 00:28:39,710 graph have a cycle? 458 00:28:43,790 --> 00:28:47,670 No, this graph is cycle free. 459 00:28:47,670 --> 00:28:51,880 I've just gotten rid of all the possible cycles. 460 00:28:51,880 --> 00:28:56,670 How can you tell if a graph is cycle free? 461 00:28:56,670 --> 00:28:57,500 Various ways. 462 00:28:57,500 --> 00:29:00,550 I think the most elegant one is to say if every edge is by 463 00:29:00,550 --> 00:29:03,740 itself a cut-set, then the graph is cycle free. 464 00:29:03,740 --> 00:29:08,670 If I remove any edge, then I decompose the 465 00:29:08,670 --> 00:29:11,010 graph into two parts. 466 00:29:11,010 --> 00:29:12,930 And that's clearly true of this graph. 467 00:29:12,930 --> 00:29:15,670 We don't really have to test the half edges. 468 00:29:15,670 --> 00:29:18,160 In fact, this is kind of the defining 469 00:29:18,160 --> 00:29:20,440 concept of a state space. 470 00:29:20,440 --> 00:29:22,480 A state space is kind of a cut between the 471 00:29:22,480 --> 00:29:24,320 past and the future. 472 00:29:24,320 --> 00:29:27,980 When we asked about the minimal state space, we asked 473 00:29:27,980 --> 00:29:31,830 what's the minimal dimension of the information, the state, 474 00:29:31,830 --> 00:29:36,730 that we need to pass from the past to the future? 475 00:29:36,730 --> 00:29:39,140 And so states correspond to cuts. 476 00:29:39,140 --> 00:29:45,120 If we make each state space, if we consider it to be a 477 00:29:45,120 --> 00:29:53,830 single variable, then we get just a chain graph on a 478 00:29:53,830 --> 00:29:57,890 sequential time axis, as is conventional in system theory, 479 00:29:57,890 --> 00:30:02,100 as your conventional integer-time axis for discrete 480 00:30:02,100 --> 00:30:03,310 time systems. 481 00:30:03,310 --> 00:30:06,040 In this case, it only has a finite number of times where 482 00:30:06,040 --> 00:30:07,380 anything happens. 483 00:30:07,380 --> 00:30:09,690 But you can think of this as part of the 484 00:30:09,690 --> 00:30:11,010 infinite set of integers. 485 00:30:11,010 --> 00:30:13,150 And so this is everything that happens. 486 00:30:13,150 --> 00:30:15,030 And a trellis is always going to be very boring. 487 00:30:15,030 --> 00:30:17,600 It's always going to look like this. 488 00:30:17,600 --> 00:30:22,270 But the advantage of now we have this cycle-free graph 489 00:30:22,270 --> 00:30:26,800 realization, we'll find out that we can do exact decoding 490 00:30:26,800 --> 00:30:29,870 of cycle-free realizations using the sum-product 491 00:30:29,870 --> 00:30:35,470 algorithm, or the min-sum algorithm, which in this case 492 00:30:35,470 --> 00:30:37,830 kind of reduces to the Viterbi algorithm with 493 00:30:37,830 --> 00:30:40,820 an asterisk on it. 494 00:30:40,820 --> 00:30:46,160 And the cost of doing that is that now have to instead of 495 00:30:46,160 --> 00:30:48,370 considering two binary variables here, I have to 496 00:30:48,370 --> 00:30:52,500 consider a single variable that has four possible states. 497 00:30:52,500 --> 00:30:56,560 So basically, I'm going to be carrying messages which are 498 00:30:56,560 --> 00:31:01,370 vectors indexed by the state space across here. 499 00:31:01,370 --> 00:31:03,970 And in this case, the vector is going to have to have four 500 00:31:03,970 --> 00:31:07,420 elements, which are going to be likelihoods or metrics or 501 00:31:07,420 --> 00:31:10,140 weights of some kind. 502 00:31:10,140 --> 00:31:13,510 It's basically telling me what's the weight of each of 503 00:31:13,510 --> 00:31:15,940 the four possible survivors, if you can think of the 504 00:31:15,940 --> 00:31:17,570 trellis that goes with this. 505 00:31:17,570 --> 00:31:18,820 Or here, I'm going to need to carry a 506 00:31:18,820 --> 00:31:20,550 vector with 8 elements. 507 00:31:20,550 --> 00:31:27,960 So it takes more to specify a vector with 8 elements than 3 508 00:31:27,960 --> 00:31:30,920 vectors each with two elements. 509 00:31:30,920 --> 00:31:35,220 So by aggregating these, I've created a sort of exponential 510 00:31:35,220 --> 00:31:40,760 complexity situation as these state spaces get very big. 511 00:31:40,760 --> 00:31:43,048 But on the other hand, I get cycle freedom. 512 00:31:43,048 --> 00:31:43,924 All right? 513 00:31:43,924 --> 00:31:44,362 Yeah. 514 00:31:44,362 --> 00:31:46,270 AUDIENCE: There's no real difference between this and 515 00:31:46,270 --> 00:31:47,520 what you had before, right? 516 00:31:51,260 --> 00:31:53,040 PROFESSOR: As President Clinton might have said, it 517 00:31:53,040 --> 00:31:57,405 depends what the definition of "real" is. 518 00:31:57,405 --> 00:31:59,920 What do you mean by real difference? 519 00:31:59,920 --> 00:32:01,780 I mean, sure. 520 00:32:01,780 --> 00:32:07,670 We think of something u1, u2, you can think of that as two 521 00:32:07,670 --> 00:32:11,190 binary variables or a single quaternary variable. 522 00:32:11,190 --> 00:32:13,540 Is there any real difference between that? 523 00:32:13,540 --> 00:32:15,470 Well, when we actually go to the implement decoding 524 00:32:15,470 --> 00:32:17,970 algorithms, we'll find there is a real difference in which 525 00:32:17,970 --> 00:32:20,260 attitude you take. 526 00:32:20,260 --> 00:32:25,110 But mathematically, it's a distinction with hardly any 527 00:32:25,110 --> 00:32:25,420 difference. 528 00:32:25,420 --> 00:32:28,090 AUDIENCE: So then there's no fundamental difference between 529 00:32:28,090 --> 00:32:31,110 the cycle freeness and the cycle-- 530 00:32:31,110 --> 00:32:34,055 PROFESSOR: No, you'll see this real difference-- 531 00:32:36,570 --> 00:32:39,370 there is another real, significant difference. 532 00:32:39,370 --> 00:32:41,662 If I drew this as two binary variables, I have 533 00:32:41,662 --> 00:32:42,840 a graph with cycles. 534 00:32:42,840 --> 00:32:47,840 And we'll find that I can't do exact decoding with a 535 00:32:47,840 --> 00:32:51,960 sum-product algorithm on a graph with cycles. 536 00:32:51,960 --> 00:32:55,720 So if I tried to apply that algorithm to this, I'd sort of 537 00:32:55,720 --> 00:32:58,490 iterate around this little cycle. 538 00:32:58,490 --> 00:33:02,360 Whereas if I agglomerate them into a single variable, I get 539 00:33:02,360 --> 00:33:03,630 rid of that behavior. 540 00:33:03,630 --> 00:33:08,940 And I can summarize everything in four values. 541 00:33:08,940 --> 00:33:11,344 So in that sense, it's a huge difference. 542 00:33:11,344 --> 00:33:14,560 AUDIENCE: Can I always transform this whole graph 543 00:33:14,560 --> 00:33:17,622 into cycle-free graph using this kind of technique? 544 00:33:17,622 --> 00:33:20,070 PROFESSOR: I didn't get the first part of your question. 545 00:33:20,070 --> 00:33:23,190 AUDIENCE: Can I always transform the graph with 546 00:33:23,190 --> 00:33:25,660 cycles to a cycle-free graph using this-- 547 00:33:25,660 --> 00:33:26,750 combining those-- 548 00:33:26,750 --> 00:33:27,700 PROFESSOR: Yeah, OK. 549 00:33:27,700 --> 00:33:35,180 So this agglomeration technique of drawing lines 550 00:33:35,180 --> 00:33:38,460 around sub-graphs and then considering everything inside 551 00:33:38,460 --> 00:33:43,480 there to be a constraint, and all the variables coming out-- 552 00:33:43,480 --> 00:33:46,900 well, it depends on their topology, but I can group the 553 00:33:46,900 --> 00:33:49,420 variables coming out however I want-- 554 00:33:49,420 --> 00:33:51,161 yes, I can always do that. 555 00:33:51,161 --> 00:33:54,107 AUDIENCE: So we can always do that for things 556 00:33:54,107 --> 00:33:56,688 [UNINTELLIGIBLE] 557 00:33:56,688 --> 00:33:58,680 transform the graph into cycle free. 558 00:33:58,680 --> 00:34:00,180 PROFESSOR: Right. 559 00:34:00,180 --> 00:34:08,020 But we may find that to make it cycle free, I then have to 560 00:34:08,020 --> 00:34:11,940 aggregate all the edges into a single edge between any two 561 00:34:11,940 --> 00:34:13,480 parts of the graph. 562 00:34:13,480 --> 00:34:15,799 And that may radically increase the complexity. 563 00:34:15,799 --> 00:34:20,530 And I'll give you several examples of that today. 564 00:34:20,530 --> 00:34:24,500 So these little fine points all of a sudden loom large 565 00:34:24,500 --> 00:34:27,650 when we actually come to build something. 566 00:34:27,650 --> 00:34:30,699 Let me give you another example of that. 567 00:34:30,699 --> 00:34:33,110 A very fine example would be to say, what does 568 00:34:33,110 --> 00:34:35,639 sectionalization consist of? 569 00:34:35,639 --> 00:34:37,320 We talked about sectionalization. 570 00:34:37,320 --> 00:34:44,120 Suppose we want to get a trellis graph for a 4-section 571 00:34:44,120 --> 00:34:48,313 trellis, where we take pairs of variables. 572 00:34:48,313 --> 00:34:52,230 Well, the graph realization of that is simply obtained by 573 00:34:52,230 --> 00:34:59,450 agglomerating pairs of these blocks, like that. 574 00:34:59,450 --> 00:35:05,250 OK, so let me do that, and then let me see what I've got. 575 00:35:05,250 --> 00:35:08,220 I've now got here, a constraint 576 00:35:08,220 --> 00:35:10,580 that affects two variables. 577 00:35:10,580 --> 00:35:16,100 Well, first of all, I've now got only two visible bits in 578 00:35:16,100 --> 00:35:18,430 each of the state variables. 579 00:35:18,430 --> 00:35:21,060 Here I have u1, u2, here I have u2, u3, 580 00:35:21,060 --> 00:35:22,960 here i have u3, u4. 581 00:35:22,960 --> 00:35:28,830 So we get rid of some of this state complexity. 582 00:35:28,830 --> 00:35:31,385 We did this trick in another way before, by 583 00:35:31,385 --> 00:35:33,600 sectionalization. 584 00:35:33,600 --> 00:35:36,240 So we get rid of a lot of state spaces, including ones 585 00:35:36,240 --> 00:35:39,140 that are big. 586 00:35:39,140 --> 00:35:40,590 Let's see, what do we have here? 587 00:35:40,590 --> 00:35:46,160 This is now a code which has got two bits coming in, two 588 00:35:46,160 --> 00:35:46,960 bits coming out. 589 00:35:46,960 --> 00:35:48,500 It's a 4, 2 code. 590 00:35:48,500 --> 00:35:51,690 It's basically controlled by u1 and u2. 591 00:35:51,690 --> 00:35:54,670 Probably y0 and y1 are some simple linear 592 00:35:54,670 --> 00:35:57,320 function of u1 and u2. 593 00:35:57,320 --> 00:36:00,300 So this is a 4, 2 constraint code. 594 00:36:00,300 --> 00:36:05,870 It controls these two bits and these two state bits, these 595 00:36:05,870 --> 00:36:08,100 two symbol bits, and these two state bits. 596 00:36:08,100 --> 00:36:09,340 What is this over here now? 597 00:36:09,340 --> 00:36:13,530 We've got 2, 2, 2. 598 00:36:13,530 --> 00:36:18,510 So this is going to be a code of length 6. 599 00:36:18,510 --> 00:36:19,940 And what dimension? 600 00:36:19,940 --> 00:36:23,090 This is, remember, u2, u3. 601 00:36:23,090 --> 00:36:26,880 So this whole behavior here is affected by u1, u2, and u3, it 602 00:36:26,880 --> 00:36:28,600 has dimension 3. 603 00:36:28,600 --> 00:36:32,200 Or there's a shortcut I can do here, because this is a 604 00:36:32,200 --> 00:36:34,160 self-dual code, all of these little codes 605 00:36:34,160 --> 00:36:36,730 are going to be self-dual. 606 00:36:36,730 --> 00:36:38,830 Well, they're not going to be self-dual, but they're going 607 00:36:38,830 --> 00:36:41,250 to be rate 1/2. 608 00:36:41,250 --> 00:36:44,800 Half as many bits here as they do here. 609 00:36:47,410 --> 00:36:52,570 But that's another duality theorem that we won't prove. 610 00:36:52,570 --> 00:36:57,270 OK, and symmetrically, we get another code here, which is a 611 00:36:57,270 --> 00:37:01,350 6, 3 code, and another one here. 612 00:37:01,350 --> 00:37:03,940 So we can do this. 613 00:37:03,940 --> 00:37:07,210 The reason I call this styles of realization is there's 614 00:37:07,210 --> 00:37:09,515 obviously a lot of freedom in how we want 615 00:37:09,515 --> 00:37:11,110 to depict the code. 616 00:37:11,110 --> 00:37:14,890 And depending on how we depict it, when we get to decoding 617 00:37:14,890 --> 00:37:17,720 algorithms, it may affect the complexity of the algorithms. 618 00:37:17,720 --> 00:37:20,380 So we want to find a nice way of depicting it. 619 00:37:20,380 --> 00:37:23,693 AUDIENCE: I don't understand why you-- so you want the ... 620 00:37:23,693 --> 00:37:25,330 freedom ... is [INAUDIBLE], because you would only have 621 00:37:25,330 --> 00:37:26,470 two [UNINTELLIGIBLE] to be used [UNINTELLIGIBLE]. 622 00:37:26,470 --> 00:37:28,922 And then you have two [UNINTELLIGIBLE] 623 00:37:28,922 --> 00:37:30,320 u1 and u2. 624 00:37:30,320 --> 00:37:31,730 [UNINTELLIGIBLE] u2 and 3. 625 00:37:31,730 --> 00:37:34,180 PROFESSOR: There are two bits here, two bits 626 00:37:34,180 --> 00:37:37,080 here, two bits here. 627 00:37:37,080 --> 00:37:37,900 AUDIENCE: But that's 6. 628 00:37:37,900 --> 00:37:38,720 That's 6. 629 00:37:38,720 --> 00:37:39,050 [INAUDIBLE] 630 00:37:39,050 --> 00:37:40,290 PROFESSOR: Weight 6, all right. 631 00:37:40,290 --> 00:37:41,756 AUDIENCE: [INAUDIBLE] freedom that you talk about. 632 00:37:41,756 --> 00:37:43,780 PROFESSOR: How many possibilities are there for 633 00:37:43,780 --> 00:37:44,640 these 6 bits? 634 00:37:44,640 --> 00:37:47,280 I've got to consider all possible combinations of u1, 635 00:37:47,280 --> 00:37:51,810 u2, and u3 to drive what's happening in these two times. 636 00:37:54,712 --> 00:37:58,940 If I go back here and look at those two times, I see that 637 00:37:58,940 --> 00:38:03,220 there are three generators that I've got to consider, 638 00:38:03,220 --> 00:38:04,890 these three. 639 00:38:04,890 --> 00:38:08,930 All right, so for all eight possible linear combinations 640 00:38:08,930 --> 00:38:12,250 of those three generators, I'll get different 641 00:38:12,250 --> 00:38:14,330 combinations of these six bits here. 642 00:38:22,650 --> 00:38:25,030 And I can go further. 643 00:38:25,030 --> 00:38:29,050 Actually, at the end of the day, I concluded here, we said 644 00:38:29,050 --> 00:38:32,720 the idea of sectionalization was to sectionalize as far as 645 00:38:32,720 --> 00:38:36,820 possible without increasing the branch complexity, which 646 00:38:36,820 --> 00:38:40,550 we've now translated into this constraint code complexity. 647 00:38:40,550 --> 00:38:45,310 And so it's even better to just keep aggregating. 648 00:38:45,310 --> 00:38:47,760 Consider the first half. 649 00:38:47,760 --> 00:38:50,740 And this is 1, 2, 3, 4, two bits there is 650 00:38:50,740 --> 00:38:53,800 still only a 6, 3 code. 651 00:38:53,800 --> 00:38:58,390 And the second half is still only a 6, 3 code. 652 00:38:58,390 --> 00:39:02,050 When we get to the sum-product algorithm, that means we have 653 00:39:02,050 --> 00:39:04,560 to compute eight things when we get to this node. 654 00:39:07,540 --> 00:39:10,530 Maybe it's a slightly more complicated thing, but as long 655 00:39:10,530 --> 00:39:12,856 as we keep the dimension down to 3, we're only going to have 656 00:39:12,856 --> 00:39:15,260 to compute eight things. 657 00:39:15,260 --> 00:39:18,790 So we haven't really increased the decoding complexity at all 658 00:39:18,790 --> 00:39:20,170 by doing this. 659 00:39:20,170 --> 00:39:25,753 So that we consider to be our best sectionalization. 660 00:39:25,753 --> 00:39:30,310 And if we tried to aggregate these two, what would we get? 661 00:39:30,310 --> 00:39:34,020 We'd simply get the eight bits here, we get a constraint that 662 00:39:34,020 --> 00:39:37,500 says it's an 8, 4 constraint, it says these eight bits have 663 00:39:37,500 --> 00:39:40,660 got to be in the code. 664 00:39:40,660 --> 00:39:43,070 So we only go that far, because that would increase 665 00:39:43,070 --> 00:39:46,020 the complexity to 16. 666 00:39:46,020 --> 00:39:49,560 All right, so that's our minimal trellis. 667 00:39:49,560 --> 00:39:54,040 We call it a two-section trellis where the symbol bits 668 00:39:54,040 --> 00:39:57,770 have been grouped into 4-tuples. 669 00:39:57,770 --> 00:40:01,560 And for decoding, that's the simplest one. 670 00:40:07,150 --> 00:40:10,700 So you see what kind of games we can play here. 671 00:40:10,700 --> 00:40:11,000 Yeah? 672 00:40:11,000 --> 00:40:13,043 AUDIENCE: At the beginning of this process, you started at 673 00:40:13,043 --> 00:40:15,540 the [UNINTELLIGIBLE]. 674 00:40:15,540 --> 00:40:21,200 PROFESSOR: Well, you remember in more detail how we 675 00:40:21,200 --> 00:40:23,660 concluded this was the optimal sectionalization. 676 00:40:23,660 --> 00:40:26,265 We looked at time 3, in particular. 677 00:40:26,265 --> 00:40:28,750 And we said there are three generators that are 678 00:40:28,750 --> 00:40:29,890 active at time 3. 679 00:40:29,890 --> 00:40:33,150 So I'm going to expand that as far as I can without bringing 680 00:40:33,150 --> 00:40:34,850 any more generators into the picture. 681 00:40:34,850 --> 00:40:38,370 So I can't expand it over here, because I'll hit u4. 682 00:40:38,370 --> 00:40:41,220 But I can expand it as far as I want over here. 683 00:40:41,220 --> 00:40:42,230 And that's the way we sectionalize. 684 00:40:42,230 --> 00:40:44,920 AUDIENCE: So do you always start with [UNINTELLIGIBLE]? 685 00:40:44,920 --> 00:40:47,570 PROFESSOR: Well, there's some art in this. 686 00:40:47,570 --> 00:40:51,480 Here it's kind of obvious where to do it. 687 00:40:51,480 --> 00:40:55,073 And I give two heuristic algorithms in the notes on 688 00:40:55,073 --> 00:40:59,020 sectionalization, which we'll come up with basically the 689 00:40:59,020 --> 00:41:00,050 same thing. 690 00:41:00,050 --> 00:41:04,320 But for simple codes, you can just eyeball it. 691 00:41:04,320 --> 00:41:09,220 OK, so let's see. 692 00:41:09,220 --> 00:41:12,630 That's another general and very useful style of 693 00:41:12,630 --> 00:41:13,105 realization. 694 00:41:13,105 --> 00:41:16,690 It's a trellis realization, or a sectionalized trellis 695 00:41:16,690 --> 00:41:17,590 realization. 696 00:41:17,590 --> 00:41:20,780 Is there anything else I wanted to say about that? 697 00:41:20,780 --> 00:41:24,830 We can do it so that we get minimal branch and state 698 00:41:24,830 --> 00:41:28,190 spaces, or constraint complexities. 699 00:41:28,190 --> 00:41:32,640 It's cycle free if we aggregate the states into a 700 00:41:32,640 --> 00:41:34,600 single state space. 701 00:41:34,600 --> 00:41:37,530 Sectionalization, we talked about that. 702 00:41:37,530 --> 00:41:41,050 OK, so let me leave that up, because we're of course not 703 00:41:41,050 --> 00:41:42,300 done with that. 704 00:41:44,620 --> 00:41:48,100 Now let me talk about some general properties of graph 705 00:41:48,100 --> 00:41:49,350 realizations. 706 00:42:00,610 --> 00:42:04,190 And the most important thing I'm going to talk about here 707 00:42:04,190 --> 00:42:06,170 is the cut-set bound. 708 00:42:09,920 --> 00:42:15,950 I want to get across the idea that a graph really captures 709 00:42:15,950 --> 00:42:18,360 dependency relationships. 710 00:42:18,360 --> 00:42:22,500 If variables are incident on the same node, they're 711 00:42:22,500 --> 00:42:23,800 obviously all dependent. 712 00:42:23,800 --> 00:42:27,460 Or if you think of larger sets of variables aggregated in 713 00:42:27,460 --> 00:42:30,040 this way, then they're somehow dependent. 714 00:42:30,040 --> 00:42:32,880 Variables that are very far away on the graph are less 715 00:42:32,880 --> 00:42:35,510 dependent on each other than ones that are close to each 716 00:42:35,510 --> 00:42:36,240 other on the graph. 717 00:42:36,240 --> 00:42:40,160 This is all kind of vague and woolly statements. 718 00:42:40,160 --> 00:42:44,190 Let's see if we can make them more concrete. 719 00:42:44,190 --> 00:42:48,830 First of all, there's a very simple but important 720 00:42:48,830 --> 00:42:52,598 observation that -- 721 00:42:52,598 --> 00:42:54,050 I'll put it this way-- 722 00:42:54,050 --> 00:43:03,215 disconnected if and only if independent. 723 00:43:03,215 --> 00:43:05,600 What does this mean? 724 00:43:05,600 --> 00:43:10,930 If I have a disconnected graph, let's suppose I have 725 00:43:10,930 --> 00:43:14,350 one graph over here with certain external variables. 726 00:43:14,350 --> 00:43:19,420 Let me aggregate them all into y1 and some constraint 1 over 727 00:43:19,420 --> 00:43:25,380 here, and a completely separate graph, some different 728 00:43:25,380 --> 00:43:29,320 code over here, c2, constraining some separate set 729 00:43:29,320 --> 00:43:30,570 of variables, y2. 730 00:43:33,990 --> 00:43:37,550 OK, that's a disconnected graph realization. 731 00:43:37,550 --> 00:43:40,060 It has to look like that, right? 732 00:43:40,060 --> 00:43:44,115 So we're aggregating the two disconnected halves. 733 00:43:44,115 --> 00:43:45,590 All right, what can I say? 734 00:43:45,590 --> 00:43:48,075 What code does this graph realize? 735 00:43:52,570 --> 00:43:57,960 This graph, the code that it realizes is simply the 736 00:43:57,960 --> 00:44:00,520 Cartesian product of c1 and c2. 737 00:44:00,520 --> 00:44:07,380 In other words, this means the set of all pairs, c1, c2, or I 738 00:44:07,380 --> 00:44:08,650 should say y1, y2-- 739 00:44:11,840 --> 00:44:19,030 y1, y2 such that y1 is in code 1 and y2 is in code 2. 740 00:44:22,000 --> 00:44:27,290 That is the set of all y1, y2 that satisfy all these 741 00:44:27,290 --> 00:44:28,480 constraints, right? 742 00:44:28,480 --> 00:44:31,810 So it's a behavioral realization of a Cartesian 743 00:44:31,810 --> 00:44:33,060 product code. 744 00:44:37,650 --> 00:44:43,510 Now this is really a notion of independence, right? 745 00:44:43,510 --> 00:44:49,100 This independently realizes a code word from c1 in this part 746 00:44:49,100 --> 00:44:51,970 and a code word from c2 in this part. 747 00:44:51,970 --> 00:44:53,850 What would the generator matrix look like? 748 00:44:53,850 --> 00:44:57,810 The generator matrix from this would look like a generator 749 00:44:57,810 --> 00:45:04,100 for c1, 0, this is a 0, and a generator for c2. 750 00:45:04,100 --> 00:45:06,720 So there's a generator matrix notion of independence. 751 00:45:06,720 --> 00:45:09,030 In other words, these are just two independent codes that for 752 00:45:09,030 --> 00:45:12,790 some reason, we choose to regard as one code. 753 00:45:12,790 --> 00:45:17,190 And the same thing is true that if we have a Cartesian 754 00:45:17,190 --> 00:45:21,160 product code, simply a code made up of two independent 755 00:45:21,160 --> 00:45:24,990 components, then we can always realize it in this way, right? 756 00:45:24,990 --> 00:45:28,550 We just realize c1, realize c2. 757 00:45:28,550 --> 00:45:32,440 So that's elementary, but it begins to get across the idea 758 00:45:32,440 --> 00:45:36,740 that graph properties have to do with dependence properties. 759 00:45:36,740 --> 00:45:40,440 And here's the most radical and simple form of that. 760 00:45:40,440 --> 00:45:43,330 If we have a disconnected graph, then it really realizes 761 00:45:43,330 --> 00:45:44,580 two independent codes. 762 00:45:48,350 --> 00:45:53,840 Now, more important than that is the cut-set bound. 763 00:46:01,100 --> 00:46:02,350 What is a cut-set? 764 00:46:05,870 --> 00:46:07,230 It's a set of edges. 765 00:46:10,270 --> 00:46:13,220 Actually, in graph theory, it's defined in various ways. 766 00:46:13,220 --> 00:46:15,070 It could be vertices, it could be edges. 767 00:46:15,070 --> 00:46:19,830 Here, we're going to say it's a set of edges whose removal 768 00:46:19,830 --> 00:46:21,080 disconnects the graph. 769 00:46:31,180 --> 00:46:33,670 Probably there are people in this room who know more graph 770 00:46:33,670 --> 00:46:34,400 theory than I do? 771 00:46:34,400 --> 00:46:39,300 Does anyone want to quibble or refine that in any way? 772 00:46:39,300 --> 00:46:42,730 That's my idea of what a cut-set is. 773 00:46:42,730 --> 00:46:44,225 Any elaboration? 774 00:46:44,225 --> 00:46:45,155 No? 775 00:46:45,155 --> 00:46:47,020 All right. 776 00:46:47,020 --> 00:46:54,110 All right, so we have some large graph realization. 777 00:46:54,110 --> 00:46:59,200 It's got vertices which I always draw as blocks, its 778 00:46:59,200 --> 00:47:05,032 constraint codes, they have various interrelationships. 779 00:47:05,032 --> 00:47:07,710 I'm just showing one. 780 00:47:07,710 --> 00:47:14,770 We have various external variables that we bring out of 781 00:47:14,770 --> 00:47:17,630 that, and that's the general graphical 782 00:47:17,630 --> 00:47:19,660 realization of the code. 783 00:47:19,660 --> 00:47:22,902 What are some cut-sets in here? 784 00:47:22,902 --> 00:47:25,950 I've drawn this, I'm thinking of this as a cut set. 785 00:47:25,950 --> 00:47:35,190 If I take these two edges and remove them, then I've 786 00:47:35,190 --> 00:47:36,330 disconnected the graph. 787 00:47:36,330 --> 00:47:37,580 So they form a cut-set. 788 00:47:40,670 --> 00:47:44,940 I've already mentioned the very close connection between 789 00:47:44,940 --> 00:47:49,060 cut-sets and cycle-free graphs, which is the graph is 790 00:47:49,060 --> 00:47:53,090 cycle-free if and only if by removing any single edge, I 791 00:47:53,090 --> 00:47:54,900 disconnect the graph. 792 00:47:54,900 --> 00:47:57,910 So every single edge is itself a cut-set. 793 00:47:57,910 --> 00:48:01,430 That's a very elegant characterization. 794 00:48:01,430 --> 00:48:03,230 All right, so I'm thinking of a cut-set. 795 00:48:03,230 --> 00:48:08,710 I'm going to write it as chi for scissors. 796 00:48:08,710 --> 00:48:12,050 And it's a set of edges. 797 00:48:12,050 --> 00:48:14,850 So in our cases, what are the edges? 798 00:48:14,850 --> 00:48:18,970 It's a set of state spaces for some index set. 799 00:48:21,700 --> 00:48:24,920 OK, so I'm going to select some minimal set of state 800 00:48:24,920 --> 00:48:29,510 spaces that disconnects the graph. 801 00:48:29,510 --> 00:48:31,670 All my edges are internal variables. 802 00:48:31,670 --> 00:48:35,100 We don't really need to worry about these half edges out 803 00:48:35,100 --> 00:48:38,430 here, as I've said before. 804 00:48:38,430 --> 00:48:42,620 They're always going to trivially separate their 805 00:48:42,620 --> 00:48:45,620 variables from the rest of the graph, and it's just tedious 806 00:48:45,620 --> 00:48:47,690 to try to keep them in the explanation. 807 00:48:47,690 --> 00:48:51,490 So we're only talking about state-edges here. 808 00:48:51,490 --> 00:48:58,370 All right, once I've done that, let me now do my 809 00:48:58,370 --> 00:48:59,620 agglomeration trick. 810 00:49:06,330 --> 00:49:13,070 Sorry, I'm trying to draw a dotted line around a part of 811 00:49:13,070 --> 00:49:17,930 the graph that I'm going to leave outside the external 812 00:49:17,930 --> 00:49:20,220 variables that I still want to be external. 813 00:49:20,220 --> 00:49:26,570 I'm going to leave the state variables, which I'm going to 814 00:49:26,570 --> 00:49:28,120 remember were once connected. 815 00:49:28,120 --> 00:49:30,930 And let me reconnect them now. 816 00:49:30,930 --> 00:49:35,150 So I'm taking my original graph and I'm dividing it 817 00:49:35,150 --> 00:49:38,550 according to this cut-set into two parts. 818 00:49:41,230 --> 00:49:45,390 And I'm arbitrarily going to call this the past, p-- 819 00:49:45,390 --> 00:49:47,720 arbitrarily but suggestively-- 820 00:49:47,720 --> 00:49:48,580 and the future, f. 821 00:49:48,580 --> 00:49:52,490 And it doesn't matter which one I call p and which one f. 822 00:49:55,654 --> 00:50:01,600 And I'm going to redraw this, so this is just a re-drawing 823 00:50:01,600 --> 00:50:05,260 of that in a nicer form. 824 00:50:05,260 --> 00:50:09,000 I'm going to aggregate all of these variables over here. 825 00:50:09,000 --> 00:50:12,140 And I'm going to call that the set of past external 826 00:50:12,140 --> 00:50:14,855 variables, y projected on the past. 827 00:50:17,365 --> 00:50:20,930 And similarly over here, I'll get the set of all external 828 00:50:20,930 --> 00:50:24,080 variables projected on the future. 829 00:50:24,080 --> 00:50:27,680 It was just the ones that occur when I make this kind of 830 00:50:27,680 --> 00:50:29,840 cut through state edges. 831 00:50:29,840 --> 00:50:32,750 Some of the external variables wind up in the past side, and 832 00:50:32,750 --> 00:50:34,870 some wind up connected to the future side. 833 00:50:34,870 --> 00:50:38,810 But there are obviously none connected to both because the 834 00:50:38,810 --> 00:50:39,860 definition of a cut-set. 835 00:50:39,860 --> 00:50:42,320 We've disconnected the graph. 836 00:50:42,320 --> 00:50:43,660 All right, so there's the past. 837 00:50:43,660 --> 00:50:45,620 There's the future. 838 00:50:45,620 --> 00:50:50,050 And here are the state variables. 839 00:50:50,050 --> 00:50:55,190 Now I'm going to define these all together as a super-state 840 00:50:55,190 --> 00:51:00,250 space as I did in the trellis graph over there. 841 00:51:00,250 --> 00:51:05,770 So let's see, here I labeled the edges by a set of states. 842 00:51:05,770 --> 00:51:10,690 But what I mean here is that the super-state space is just 843 00:51:10,690 --> 00:51:13,385 the product of the component state spaces. 844 00:51:16,100 --> 00:51:29,570 It's elements s chi are just the set of sj, j, and sigma-j, 845 00:51:29,570 --> 00:51:32,300 is that clear? 846 00:51:32,300 --> 00:51:35,040 In other words, I have a vector of states here. 847 00:51:35,040 --> 00:51:39,820 And I consider the super-state space to be just the 848 00:51:39,820 --> 00:51:40,400 Cartesian-- 849 00:51:40,400 --> 00:51:42,340 this, again, is the Cartesian product of 850 00:51:42,340 --> 00:51:45,260 all the state spaces. 851 00:51:45,260 --> 00:51:49,930 OK, and then I have constraints. 852 00:51:49,930 --> 00:51:54,160 So we'll call this the aggregate past constraint, and 853 00:51:54,160 --> 00:51:55,560 the aggregate future constraint. 854 00:51:58,580 --> 00:52:01,050 This constrains these variables in this state. 855 00:52:01,050 --> 00:52:03,471 This constrains these variables in that state. 856 00:52:06,060 --> 00:52:08,790 OK, so that's where figure 5 comes from. 857 00:52:08,790 --> 00:52:12,500 Every year, people say, how did you get figure 5? 858 00:52:12,500 --> 00:52:13,880 That's how I get figure 5. 859 00:52:13,880 --> 00:52:15,400 Is there any confusion about that? 860 00:52:18,320 --> 00:52:22,190 Today I've been talking about agglomeration quite a bit, so 861 00:52:22,190 --> 00:52:25,220 maybe this year it'll come through better 862 00:52:25,220 --> 00:52:27,348 than in past years. 863 00:52:27,348 --> 00:52:28,286 AUDIENCE: [INAUDIBLE] 864 00:52:28,286 --> 00:52:30,631 that we see projected on it. 865 00:52:30,631 --> 00:52:33,450 Or is it just-- 866 00:52:33,450 --> 00:52:36,640 PROFESSOR: It's just the aggregate of all these 867 00:52:36,640 --> 00:52:38,490 constraints. 868 00:52:38,490 --> 00:52:39,770 What does this say? 869 00:52:39,770 --> 00:52:42,830 It's the set of all the possible values of these 870 00:52:42,830 --> 00:52:46,120 variables, and these variables that can actually occur. 871 00:52:46,120 --> 00:52:49,670 That forms a linear space which we call little code. 872 00:52:49,670 --> 00:52:53,580 Anything consistent with that satisfies this constraint. 873 00:52:53,580 --> 00:52:55,270 Anything else doesn't. 874 00:52:55,270 --> 00:52:57,745 AUDIENCE: [INAUDIBLE] 875 00:52:57,745 --> 00:53:04,180 for the c's projection of the code on the past? 876 00:53:04,180 --> 00:53:06,390 PROFESSOR: No. 877 00:53:06,390 --> 00:53:10,770 The projection of the code on the past would be you realize 878 00:53:10,770 --> 00:53:14,140 the code and you just project onto these variables here. 879 00:53:14,140 --> 00:53:16,450 So no, this is just an index. 880 00:53:16,450 --> 00:53:19,590 This is a constraint whose index is the past. 881 00:53:23,170 --> 00:53:27,400 OK, well, all right. 882 00:53:27,400 --> 00:53:31,590 Oh, but this looks like a trellis, like 883 00:53:31,590 --> 00:53:32,940 a two-section trellis. 884 00:53:32,940 --> 00:53:34,845 It looks very much like this thing right here. 885 00:53:37,370 --> 00:53:40,600 It has the same general form. 886 00:53:40,600 --> 00:53:46,600 And we can think of this super-state variable here, it 887 00:53:46,600 --> 00:53:50,130 sort of has the Markov property. 888 00:53:50,130 --> 00:53:55,910 It, again, tells everything about the past that is needed 889 00:53:55,910 --> 00:53:59,090 to specify what possible futures there could be. 890 00:53:59,090 --> 00:54:01,670 We're really interested in these up here, eventually. 891 00:54:01,670 --> 00:54:04,850 But this is the whole set of constraints that determine the 892 00:54:04,850 --> 00:54:07,160 futures along with-- there may be some more free 893 00:54:07,160 --> 00:54:08,760 variables over here. 894 00:54:08,760 --> 00:54:13,030 But these are the constraints we get from this half of the 895 00:54:13,030 --> 00:54:16,930 code on this half of the code and vice versa. 896 00:54:16,930 --> 00:54:23,915 So again, a cut-set is related to a Markov property. 897 00:54:28,450 --> 00:54:31,620 Let me try to state this property more explicitly. 898 00:54:36,990 --> 00:54:43,310 The code by the behavioral realization is simply the set 899 00:54:43,310 --> 00:54:44,680 of all combinations of -- 900 00:54:47,370 --> 00:54:59,460 let me say, the behavior is the set of all past states and 901 00:54:59,460 --> 00:55:03,155 futures that satisfy the constraints. 902 00:55:13,090 --> 00:55:17,960 OK, so this gives me a certain set of pasts that are 903 00:55:17,960 --> 00:55:22,850 consistent with each possible value of the state variable. 904 00:55:22,850 --> 00:55:30,180 Let's call that y projected on the past that's consistent 905 00:55:30,180 --> 00:55:33,850 with a particular value of the state variable, the 906 00:55:33,850 --> 00:55:38,280 super-state, the vector of states here. 907 00:55:38,280 --> 00:55:42,200 And these are the ones in the future that are consistent 908 00:55:42,200 --> 00:55:43,450 with that state variable. 909 00:55:46,960 --> 00:55:51,640 We define that so the set of all possible past and future 910 00:55:51,640 --> 00:55:55,362 y's is simply this Cartesian product. 911 00:55:55,362 --> 00:55:57,760 And we say that in a more graph theoretic way. 912 00:55:57,760 --> 00:56:02,990 If I specify the state here as some particular value, sx, and 913 00:56:02,990 --> 00:56:06,920 I fix that, I've really disconnected the graph. 914 00:56:10,940 --> 00:56:14,920 For a fixed sx, I can just put that over here, 915 00:56:14,920 --> 00:56:16,170 put that over here. 916 00:56:18,790 --> 00:56:22,640 And this realizes, this connected graph that realizes 917 00:56:22,640 --> 00:56:27,710 the Cartesian product of whatever -- yp of sx. 918 00:56:27,710 --> 00:56:34,890 This just comes through and affects that with yf of sx. 919 00:56:34,890 --> 00:56:38,550 So for any particular value of the state, I have a certain 920 00:56:38,550 --> 00:56:43,190 Cartesian product of pasts and futures they can occur. 921 00:56:43,190 --> 00:56:47,010 Any past that's connected with that state can be connected to 922 00:56:47,010 --> 00:56:49,790 any future that's connected with this state. 923 00:56:49,790 --> 00:56:52,520 All the possible future continuations of any 924 00:56:52,520 --> 00:56:55,650 particular past in this equivalence 925 00:56:55,650 --> 00:56:58,950 class are the same. 926 00:56:58,950 --> 00:57:01,110 This should sound very much like the kinds of things we 927 00:57:01,110 --> 00:57:03,850 were talking about when we did the state space theorem. 928 00:57:07,750 --> 00:57:11,172 And we get a sort of state space theorem out of this. 929 00:57:25,070 --> 00:57:30,860 Just to draw this out, we kind of get a two section trellis 930 00:57:30,860 --> 00:57:40,270 realization where we have the various states in the state 931 00:57:40,270 --> 00:57:45,300 space here, s0, s1, and so forth. 932 00:57:45,300 --> 00:57:49,370 We have the pasts that are connected with 933 00:57:49,370 --> 00:57:52,025 s0, s1, and so forth. 934 00:57:59,770 --> 00:58:05,610 And we have the futures that are connected to s0, s1. 935 00:58:14,790 --> 00:58:18,790 So we could get from this, this picture of the two 936 00:58:18,790 --> 00:58:20,040 sectioned trellis. 937 00:58:22,180 --> 00:58:25,700 This really represents a set of parallel transitions here. 938 00:58:25,700 --> 00:58:30,440 These are all the possible pasts that are consistent with 939 00:58:30,440 --> 00:58:31,110 that state. 940 00:58:31,110 --> 00:58:32,560 These are all the futures that are 941 00:58:32,560 --> 00:58:33,940 consistent with that state. 942 00:58:33,940 --> 00:58:38,470 We can tie them together, and that realizes the code. 943 00:58:38,470 --> 00:58:45,150 I should say explicitly, the code now is the union over the 944 00:58:45,150 --> 00:58:54,226 states in the state space, the super-state space of the pasts 945 00:58:54,226 --> 00:58:59,135 there times the futures. 946 00:58:59,135 --> 00:59:00,570 They're consistent with the state. 947 00:59:00,570 --> 00:59:02,450 So it's a union of Cartesian products. 948 00:59:05,420 --> 00:59:08,016 And that's expressed by this diagram. 949 00:59:08,016 --> 00:59:08,965 This is one of them. 950 00:59:08,965 --> 00:59:10,550 This is another one. 951 00:59:10,550 --> 00:59:12,250 These are all of them. 952 00:59:12,250 --> 00:59:13,260 How many are there? 953 00:59:13,260 --> 00:59:16,910 The size of the super state variable, which by the way is 954 00:59:16,910 --> 00:59:20,200 the product of the sizes of the state spaces of each of 955 00:59:20,200 --> 00:59:23,150 these individual state spaces that I had coming across here. 956 00:59:27,770 --> 00:59:29,020 OK. 957 00:59:30,910 --> 00:59:35,025 What's the cut-set bound then? 958 00:59:35,025 --> 00:59:37,930 For linear codes, it's basically the same as the 959 00:59:37,930 --> 00:59:39,180 state space theorem. 960 00:59:49,700 --> 01:00:01,590 The state space theorem said that the dimension of the 961 01:00:01,590 --> 01:00:05,130 state space at any time -- well, let me consider now this 962 01:00:05,130 --> 01:00:07,462 state space, the super-state variable. 963 01:00:07,462 --> 01:00:11,050 The dimension of this super-state variable has got 964 01:00:11,050 --> 01:00:15,620 to be greater than or equal to the dimension of the code that 965 01:00:15,620 --> 01:00:21,630 we're realizing minus the dimension of the -- 966 01:00:21,630 --> 01:00:23,760 this is going to be bad notation now. 967 01:00:26,530 --> 01:00:29,850 Now I mean the sub-code, so I should have written something 968 01:00:29,850 --> 01:00:31,300 else for the constraint here. 969 01:00:36,320 --> 01:00:39,310 Minus the dimension of the sub-code whose support is 970 01:00:39,310 --> 01:00:42,550 entirely on the past, minus the dimension of the sub-code 971 01:00:42,550 --> 01:00:45,260 whose support is entirely on the future. 972 01:00:45,260 --> 01:00:46,750 And the state space theorem still 973 01:00:46,750 --> 01:00:51,285 applies for this partition. 974 01:00:51,285 --> 01:00:59,490 So that gives me a lower bound on the total dimension of the 975 01:00:59,490 --> 01:01:02,220 states in any cut-set. 976 01:01:10,060 --> 01:01:23,410 So for example, in the 8, 4 code, suppose we have any 977 01:01:23,410 --> 01:01:38,570 realization, so some graph here, and any cut-set such 978 01:01:38,570 --> 01:01:41,420 that the size of -- 979 01:01:46,440 --> 01:01:48,475 I should say the dimension. 980 01:01:48,475 --> 01:01:52,940 I really mean that there are four external variables in the 981 01:01:52,940 --> 01:01:54,730 past and in the future. 982 01:01:54,730 --> 01:01:59,340 This is dimension, dimension. 983 01:01:59,340 --> 01:02:03,410 In other words, we have four variables sticking out here, 984 01:02:03,410 --> 01:02:07,260 and four variables sticking out here, and we have a cut 985 01:02:07,260 --> 01:02:09,510 set somehow going through there. 986 01:02:12,030 --> 01:02:17,520 So our picture now is going to look something like this. 987 01:02:17,520 --> 01:02:23,050 What's the minimum possible dimension of the total 988 01:02:23,050 --> 01:02:28,350 dimension of the edges that cross this cut-set, the state 989 01:02:28,350 --> 01:02:30,240 spaces that are in this cut-set. 990 01:02:30,240 --> 01:02:31,870 We know this code very well now. 991 01:02:37,290 --> 01:02:38,450 Let's do the Muder bound. 992 01:02:38,450 --> 01:02:40,170 What's the maximum dimension? 993 01:02:40,170 --> 01:02:41,920 The dimension of the code is 4. 994 01:02:41,920 --> 01:02:47,690 What's the maximum dimension of any code that 995 01:02:47,690 --> 01:02:51,090 lives on the past? 996 01:02:51,090 --> 01:02:53,050 It's 1, because such a code has to have 997 01:02:53,050 --> 01:02:54,490 minimum distance 4. 998 01:02:54,490 --> 01:02:58,590 So it can't have dimension greater than the repetition 999 01:02:58,590 --> 01:03:00,560 code of length 4. 1000 01:03:00,560 --> 01:03:05,410 So 4 minus 1, same argument for the future, equals 2. 1001 01:03:08,550 --> 01:03:13,470 So it says that however you do it, you're going to have to 1002 01:03:13,470 --> 01:03:18,110 have at least two binary edges or one quaternary edge 1003 01:03:18,110 --> 01:03:23,390 crossing a cut-set that divides the external variables 1004 01:03:23,390 --> 01:03:24,820 into two equal sized parts. 1005 01:03:30,180 --> 01:03:32,770 So let's go further. 1006 01:03:32,770 --> 01:03:37,550 Let's suppose we want a cycle-free graph. 1007 01:03:37,550 --> 01:03:40,805 OK, if we have a cycle-free graph, then all of the 1008 01:03:40,805 --> 01:03:43,640 cut-sets are single edges. 1009 01:03:43,640 --> 01:03:49,165 All right, so in a cycle-free graph, when we make a cut, 1010 01:03:49,165 --> 01:03:50,910 it's through a single edge. 1011 01:03:50,910 --> 01:03:56,000 And we're saying the state space if there's a way you can 1012 01:03:56,000 --> 01:03:58,570 make a cut that divides this into two equal parts, the 1013 01:03:58,570 --> 01:04:02,000 state space has to have dimension 2, it has to have 1014 01:04:02,000 --> 01:04:04,800 size 4 for any cycle-free graph. 1015 01:04:12,650 --> 01:04:17,580 Similarly, if we have the 24, 12, 8 code, we've proved that 1016 01:04:17,580 --> 01:04:24,500 the minimum dimension of any central cut has 1017 01:04:24,500 --> 01:04:28,820 to be at least 6. 1018 01:04:28,820 --> 01:04:31,630 No, in the center, it has to be at least 8, right? 1019 01:04:37,870 --> 01:04:45,910 Yeah, because the dimension of the code is 12, and a sub-code 1020 01:04:45,910 --> 01:04:49,810 that has support on 12 can't have dimension more than 2. 1021 01:04:49,810 --> 01:04:52,250 This can't have dimension more than 2. 1022 01:04:52,250 --> 01:04:54,140 So that would be 8. 1023 01:04:54,140 --> 01:05:01,090 So for the 24, 12, 8 code, the set of edges that cross the 1024 01:05:01,090 --> 01:05:05,080 cut have to have total dimension at least 8. 1025 01:05:05,080 --> 01:05:08,720 And if we want it to be cycle free, then there has to be a 1026 01:05:08,720 --> 01:05:11,381 single edge that has dimension 8. 1027 01:05:14,090 --> 01:05:30,293 So the general conclusions -- 1028 01:05:37,930 --> 01:05:40,890 we first talked about trellis realizations. 1029 01:05:40,890 --> 01:05:45,940 We can generalize that any cycle-free realization-- 1030 01:05:45,940 --> 01:05:50,690 and we're going to see that we have strong motivation to keep 1031 01:05:50,690 --> 01:05:55,180 our realization cycle-free, because then we can do exact 1032 01:05:55,180 --> 01:05:58,080 maximum likelihood decoding. 1033 01:05:58,080 --> 01:06:02,610 So we'd like to have cycle-free realizations. 1034 01:06:02,610 --> 01:06:05,450 But what have we just seen? 1035 01:06:05,450 --> 01:06:23,180 We've just seen that the state complexity cannot be less than 1036 01:06:23,180 --> 01:06:40,120 that of a trellis realization with a comparable cut. 1037 01:06:40,120 --> 01:06:44,220 In other words, one that divides the past and future in 1038 01:06:44,220 --> 01:06:47,670 the same way, divides the external variables 1039 01:06:47,670 --> 01:06:51,720 into the same size. 1040 01:06:51,720 --> 01:06:55,960 OK, so we really can't beat the trellis bounds, the Muder 1041 01:06:55,960 --> 01:06:59,240 bound, in particular. 1042 01:06:59,240 --> 01:07:02,520 Now again, there's some games we can play. 1043 01:07:02,520 --> 01:07:06,365 Let's ask about the 24, 12, 8 cut. 1044 01:07:06,365 --> 01:07:11,230 We've said if we want to realize it with the cycle-free 1045 01:07:11,230 --> 01:07:16,750 realization, this says, well, if we have a cut that divides 1046 01:07:16,750 --> 01:07:23,390 the external variables into two parts of size 8, then 1047 01:07:23,390 --> 01:07:26,510 we're going to be stuck with a state space of dimension 8 and 1048 01:07:26,510 --> 01:07:28,490 a size 256. 1049 01:07:28,490 --> 01:07:31,610 No way around it. 1050 01:07:31,610 --> 01:07:36,700 Here's a slight way around it. 1051 01:07:36,700 --> 01:07:39,860 Let me hypothesize a realization 1052 01:07:39,860 --> 01:07:41,110 that looks like this. 1053 01:07:46,840 --> 01:07:50,380 This is a 14, 7 code. 1054 01:07:50,380 --> 01:07:52,490 So is this. 1055 01:07:52,490 --> 01:07:54,475 So is this. 1056 01:07:54,475 --> 01:07:59,160 We're going to divide the external variables into three 1057 01:07:59,160 --> 01:08:01,330 parts, each of size 8. 1058 01:08:07,790 --> 01:08:12,210 And we're going to have three internal state spaces, each of 1059 01:08:12,210 --> 01:08:13,780 dimension 6. 1060 01:08:13,780 --> 01:08:17,959 And we're going to constrain those by an 18, 9 code. 1061 01:08:17,959 --> 01:08:22,069 And again, these all have rate 1/2 because this code is dual. 1062 01:08:22,069 --> 01:08:24,750 Now, is there anything in what I've done that prevents this 1063 01:08:24,750 --> 01:08:26,000 kind of realization? 1064 01:08:28,710 --> 01:08:30,020 Let's test the cut-sets. 1065 01:08:30,020 --> 01:08:31,689 Where are the cut-sets in-- 1066 01:08:31,689 --> 01:08:34,330 well, A, is this cycle-free realization? 1067 01:08:34,330 --> 01:08:37,500 Yes, OK, good. 1068 01:08:37,500 --> 01:08:38,819 Let's test the cut-sets. 1069 01:08:38,819 --> 01:08:42,229 It's kind of a three-way symmetrical thing. 1070 01:08:42,229 --> 01:08:46,689 The internal cut-sets are here, here, and here. 1071 01:08:46,689 --> 01:08:52,590 Each of these cut-sets divides the coordinates into one set 1072 01:08:52,590 --> 01:08:55,255 of size 8 and one set of size 16. 1073 01:08:57,899 --> 01:09:01,550 In that case, what does the Muder bound give us? 1074 01:09:01,550 --> 01:09:04,590 In that case, 8, 16, the dimension of the 1075 01:09:04,590 --> 01:09:06,450 code is still 12. 1076 01:09:06,450 --> 01:09:10,850 The minimum dimension of a code of length 16 and minimum 1077 01:09:10,850 --> 01:09:13,370 distance 8 is 5. 1078 01:09:13,370 --> 01:09:18,010 And one of length 8 and minimum distance 8 is 1. 1079 01:09:18,010 --> 01:09:22,950 So as we did on the homework, we find that we are permitted 1080 01:09:22,950 --> 01:09:24,399 to have a state space-- 1081 01:09:24,399 --> 01:09:28,100 the minimal state space here could have dimension 6. 1082 01:09:28,100 --> 01:09:31,180 So this possibly could occur. 1083 01:09:31,180 --> 01:09:32,979 Again, we're kind of camouflaging some 1084 01:09:32,979 --> 01:09:34,300 of the state space. 1085 01:09:34,300 --> 01:09:36,870 You see there is no central state space in this. 1086 01:09:36,870 --> 01:09:40,670 There is no cut-set that partitions it into 12 and 12. 1087 01:09:40,670 --> 01:09:43,529 So by being a little bit clever, we've got it down to 1088 01:09:43,529 --> 01:09:48,069 something where instead of a 256 central state space, we've 1089 01:09:48,069 --> 01:09:52,100 got three 64 state spaces. 1090 01:09:52,100 --> 01:09:54,795 And this is what we saw on the trellis realization too, that 1091 01:09:54,795 --> 01:09:59,250 if we sectionalize into three sections, then we get a 64 1092 01:09:59,250 --> 01:10:03,180 state and a 64 state at the boundaries between the three 1093 01:10:03,180 --> 01:10:07,580 sections, but we still got the same 512 branch complexity in 1094 01:10:07,580 --> 01:10:08,540 the middle. 1095 01:10:08,540 --> 01:10:09,450 And what do you know? 1096 01:10:09,450 --> 01:10:11,550 We still have a constraint code with 1097 01:10:11,550 --> 01:10:14,090 complexity 512 here, too. 1098 01:10:14,090 --> 01:10:16,870 So if you consider that the more valid measure of 1099 01:10:16,870 --> 01:10:20,030 complexity, the minimum dimension of any constraint 1100 01:10:20,030 --> 01:10:24,360 code or branch space, then we haven't improved. 1101 01:10:24,360 --> 01:10:28,510 So subject to these qualifiers, this is a pretty 1102 01:10:28,510 --> 01:10:32,380 strong argument that by going beyond trellis realizations to 1103 01:10:32,380 --> 01:10:41,090 general cycle-free, graphical realizations, we 1104 01:10:41,090 --> 01:10:44,212 can't gain very much. 1105 01:10:44,212 --> 01:10:50,470 So the moral is to significantly reduce 1106 01:10:50,470 --> 01:10:55,206 complexity, we're going to need to go 1107 01:10:55,206 --> 01:10:56,465 to graphs with cycles. 1108 01:11:04,280 --> 01:11:07,870 We must go to graphs with cycles. 1109 01:11:15,910 --> 01:11:22,360 OK, so you don't get much for free in this life. 1110 01:11:22,360 --> 01:11:24,805 So when we get to our sum-product decoding 1111 01:11:24,805 --> 01:11:27,060 algorithm, we're going to have to apply it to graphs with 1112 01:11:27,060 --> 01:11:30,340 cycles where it's iterative, where it's not exact, where 1113 01:11:30,340 --> 01:11:34,920 it's just a lot less nice. 1114 01:11:34,920 --> 01:11:37,210 So what do we do when we go to graphs with cycles? 1115 01:11:37,210 --> 01:11:39,470 Where is the potential gain? 1116 01:11:39,470 --> 01:11:45,740 The potential gain is that now this super-state variable 1117 01:11:45,740 --> 01:11:52,680 could be made up of a number of simpler variables. 1118 01:11:52,680 --> 01:11:55,820 We've got a certain minimum dimension that we're going to 1119 01:11:55,820 --> 01:12:02,265 need any cut-set, like 8 for the central state space of the 1120 01:12:02,265 --> 01:12:03,300 Golay code. 1121 01:12:03,300 --> 01:12:09,290 But now we can spread it around over two or more 1122 01:12:09,290 --> 01:12:11,630 sub-state variables. 1123 01:12:11,630 --> 01:12:16,260 So for instance, maybe we can find something where we now 1124 01:12:16,260 --> 01:12:21,220 have a graph with cycles where we have two state spaces here. 1125 01:12:21,220 --> 01:12:23,740 Here's the cycle again. 1126 01:12:23,740 --> 01:12:28,880 And the two state spaces each have size only 16. 1127 01:12:28,880 --> 01:12:33,190 That would still satisfy this cut-set bound. 1128 01:12:33,190 --> 01:12:41,260 So we greatly reduce the size of the state spaces, which is 1129 01:12:41,260 --> 01:12:44,730 exponential in their dimension, but at the cost of 1130 01:12:44,730 --> 01:12:45,720 introducing a cycle. 1131 01:12:45,720 --> 01:12:47,560 I insist that's a cycle. 1132 01:12:47,560 --> 01:12:49,510 Let's go over to this trellis here. 1133 01:12:49,510 --> 01:12:53,930 The next thing I might talk about is tail-biting trellis 1134 01:12:53,930 --> 01:12:55,180 realizations. 1135 01:13:00,100 --> 01:13:01,370 This is very simple. 1136 01:13:01,370 --> 01:13:05,480 What's the first realization you would think of on a graph 1137 01:13:05,480 --> 01:13:07,490 with a cycle? 1138 01:13:07,490 --> 01:13:10,750 Well, here's a trellis realization. 1139 01:13:10,750 --> 01:13:12,370 It's always going to look like this. 1140 01:13:15,200 --> 01:13:18,074 And then let's just loop this around. 1141 01:13:18,074 --> 01:13:21,380 In other words, we make the last state space equal to the 1142 01:13:21,380 --> 01:13:22,450 first state space. 1143 01:13:22,450 --> 01:13:25,090 We don't require they used to be trivial state 1144 01:13:25,090 --> 01:13:27,040 spaces of size 1. 1145 01:13:27,040 --> 01:13:29,710 We allow them to have some dimension. 1146 01:13:29,710 --> 01:13:32,860 And here's a potential realization on a 1147 01:13:32,860 --> 01:13:34,110 graph with a cycle. 1148 01:13:38,150 --> 01:13:39,920 Cut-sets. 1149 01:13:39,920 --> 01:13:41,600 Where are the cut-sets in this graph? 1150 01:13:44,288 --> 01:13:47,720 Well, the cut-sets now look like that. 1151 01:13:47,720 --> 01:13:49,460 All the cut-sets involve at least two 1152 01:13:49,460 --> 01:13:52,450 edges, two state spaces. 1153 01:13:52,450 --> 01:13:54,840 OK, so we potentially might get this kind of benefit. 1154 01:13:58,330 --> 01:14:00,850 Here's an example. 1155 01:14:00,850 --> 01:14:06,660 Suppose instead, go back to our very favorite example, 1156 01:14:06,660 --> 01:14:13,680 suppose we let u2 go across here, and we bring out u3 out 1157 01:14:13,680 --> 01:14:18,220 of here, and we bring it all the way back to here. 1158 01:14:21,870 --> 01:14:24,540 Everything still good? 1159 01:14:24,540 --> 01:14:28,050 It's obviously another realization. 1160 01:14:28,050 --> 01:14:31,140 Now I've made it clear that when I draw u2 and u3 1161 01:14:31,140 --> 01:14:34,240 separately here, I've really got a cycle. 1162 01:14:34,240 --> 01:14:35,490 I've made a big cycle. 1163 01:14:38,130 --> 01:14:42,960 And now what's the number of states in this two-section 1164 01:14:42,960 --> 01:14:46,680 realization of the 8, 4 code? 1165 01:14:46,680 --> 01:14:49,090 This is a two-state tail-biting trellis, because 1166 01:14:49,090 --> 01:14:51,440 each of these is just a little binary variable. 1167 01:14:57,770 --> 01:14:59,970 So I get some kind of two-state trellis. 1168 01:14:59,970 --> 01:15:03,330 If I were actually to draw out the trellis, I would have two 1169 01:15:03,330 --> 01:15:10,086 states at time 0, two states at time 4, some kind of -- 1170 01:15:10,086 --> 01:15:11,150 what have I got-- 1171 01:15:11,150 --> 01:15:11,800 6, 3. 1172 01:15:11,800 --> 01:15:16,210 So I've got 8 branches that go back and forth, a couple of 1173 01:15:16,210 --> 01:15:17,250 parallel branches. 1174 01:15:17,250 --> 01:15:18,780 It's going to look like that. 1175 01:15:21,680 --> 01:15:26,940 Then same thing out to time 8. 1176 01:15:26,940 --> 01:15:28,090 I don't have a time 8. 1177 01:15:28,090 --> 01:15:37,000 This is really time 0 again, where I identify these two 1178 01:15:37,000 --> 01:15:37,920 state spaces. 1179 01:15:37,920 --> 01:15:45,130 This is really the same state as this, and this is the same 1180 01:15:45,130 --> 01:15:47,240 one as that. 1181 01:15:47,240 --> 01:15:50,790 So that's if I really drew out the trellis in the style I 1182 01:15:50,790 --> 01:15:55,400 originally drew trellises, this would be a 1183 01:15:55,400 --> 01:15:57,670 picture of all the-- 1184 01:15:57,670 --> 01:16:00,090 there's a 1-to-1 map between all the paths through this 1185 01:16:00,090 --> 01:16:03,970 trellis that start and end in the same state, because that's 1186 01:16:03,970 --> 01:16:07,340 what I mean when I identify the beginning and end states. 1187 01:16:07,340 --> 01:16:13,170 So if I start in this state, there's a four-way branch 1188 01:16:13,170 --> 01:16:17,510 here, another four-way branch there. 1189 01:16:17,510 --> 01:16:19,370 Is that right? 1190 01:16:19,370 --> 01:16:21,820 No, there's only a two-way branch such that I can get 1191 01:16:21,820 --> 01:16:22,745 back to the initial state. 1192 01:16:22,745 --> 01:16:26,870 So there are 8 possible paths that start here and get back 1193 01:16:26,870 --> 01:16:30,120 to the same state, 8 possible paths that start out from here 1194 01:16:30,120 --> 01:16:32,030 and get back to the same state. 1195 01:16:32,030 --> 01:16:39,820 And they together correspond 1-to-1 to all the code words. 1196 01:16:39,820 --> 01:16:42,430 OK, well, that's not a very big deal. 1197 01:16:42,430 --> 01:16:44,960 I was able to reduce a four-state trellis to a 1198 01:16:44,960 --> 01:16:46,210 two-state trellis. 1199 01:16:48,990 --> 01:16:54,145 But suppose I do the same thing with the Golay code. 1200 01:16:54,145 --> 01:16:58,080 In the Golay code, there's a very beautiful tail-biting 1201 01:16:58,080 --> 01:16:59,410 trellis realization. 1202 01:17:02,410 --> 01:17:03,710 It looks like this. 1203 01:17:03,710 --> 01:17:09,440 It has 12 sections, so maybe I won't draw all the sections. 1204 01:17:09,440 --> 01:17:19,110 It groups each of the output variables into pairs and comes 1205 01:17:19,110 --> 01:17:20,710 around like that. 1206 01:17:20,710 --> 01:17:26,860 And each of these state spaces has dimension 4, or size 16. 1207 01:17:26,860 --> 01:17:30,860 And I give the generator matrix for this in the notes. 1208 01:17:30,860 --> 01:17:42,350 So this is for the 24, 12, 8 code, and now you test me. 1209 01:17:42,350 --> 01:17:43,600 Is this a possible-- 1210 01:17:46,800 --> 01:17:48,090 use the cut-set bound. 1211 01:17:48,090 --> 01:17:50,130 See if this violates the cut-set bound anywhere. 1212 01:17:50,130 --> 01:17:52,210 Does it for this code? 1213 01:17:55,416 --> 01:17:56,620 No, it doesn't. 1214 01:17:56,620 --> 01:18:00,310 Because every cut-set, no matter how I draw it, we're 1215 01:18:00,310 --> 01:18:03,950 going to get two edges, each with dimension 4, adding up to 1216 01:18:03,950 --> 01:18:06,580 a super-state of dimension 8. 1217 01:18:06,580 --> 01:18:12,430 And the state spaces at all the even times, notice I've 1218 01:18:12,430 --> 01:18:15,900 sectionalized here, so I only need to look at even times. 1219 01:18:15,900 --> 01:18:20,420 The state spaces at all even times in this code could have 1220 01:18:20,420 --> 01:18:24,430 dimensions as small as 8. 1221 01:18:24,430 --> 01:18:27,760 Remember, at the odd times, they go up to 9. 1222 01:18:27,760 --> 01:18:30,090 And these are all little-- 1223 01:18:30,090 --> 01:18:33,830 4, 4-- these are all little 10, 5 codes. 1224 01:18:33,830 --> 01:18:37,612 So the branch complexity is only 32. 1225 01:18:37,612 --> 01:18:40,520 So this is very much simpler than any of the conventional 1226 01:18:40,520 --> 01:18:42,370 trellises that I've-- 1227 01:18:42,370 --> 01:18:44,480 than the minimal conventional trellis that we were able to 1228 01:18:44,480 --> 01:18:46,210 draw for this code. 1229 01:18:46,210 --> 01:18:49,490 Or if this funny little pinwheel with three arms on 1230 01:18:49,490 --> 01:18:53,540 it, which is also a cycle-free realization, this code. 1231 01:18:53,540 --> 01:18:56,760 OK, so they're going to tail-biting. 1232 01:18:56,760 --> 01:19:01,080 That's a significant advance. 1233 01:19:01,080 --> 01:19:03,500 An aside note-- 1234 01:19:03,500 --> 01:19:11,040 if we just break this, it turns out we have a generator 1235 01:19:11,040 --> 01:19:16,990 matrix that has period 4, and let this go on infinitely, 1236 01:19:16,990 --> 01:19:31,310 this is a realization of a rate 1/2, 16 state, branch 1237 01:19:31,310 --> 01:19:34,810 complexity 32, as we expect-- 1238 01:19:34,810 --> 01:19:37,160 perfectly conventional, except it's 1239 01:19:37,160 --> 01:19:38,605 periodically time varying-- 1240 01:19:45,390 --> 01:19:51,310 linear convolutional code with absolutely phenomenal 1241 01:19:51,310 --> 01:19:53,470 performance. 1242 01:19:53,470 --> 01:19:57,760 Like the Golay code, it has d equals 8. 1243 01:19:57,760 --> 01:20:03,620 Therefore it has a nominal coding gain of 4, 1/2 1244 01:20:03,620 --> 01:20:05,890 times 8, or 6 dB. 1245 01:20:05,890 --> 01:20:08,860 This is with only a 16 state, rate 1/2 code. 1246 01:20:08,860 --> 01:20:10,600 Look at the ones on the table. 1247 01:20:10,600 --> 01:20:13,910 This is significantly better than that. 1248 01:20:13,910 --> 01:20:17,930 And it's effective coding gain, I forget. 1249 01:20:17,930 --> 01:20:22,720 But it's still very good, excellent, for the complexity 1250 01:20:22,720 --> 01:20:24,370 of this code. 1251 01:20:24,370 --> 01:20:28,405 So this is called the Golay convolutional code. 1252 01:20:36,710 --> 01:20:41,510 And it's an interesting example of the interplay 1253 01:20:41,510 --> 01:20:44,360 between this block coding stuff and the convolutional 1254 01:20:44,360 --> 01:20:47,060 coding stuff, in this case. 1255 01:20:47,060 --> 01:20:49,320 Actually this code was first discovered in computer 1256 01:20:49,320 --> 01:20:52,710 searches, but nobody realized how wonderful it was. 1257 01:20:52,710 --> 01:20:55,170 And it was only when its connection with this Golay 1258 01:20:55,170 --> 01:20:58,990 code was recognized that people realized this was 1259 01:20:58,990 --> 01:21:03,530 really a special code, both from a performance versus 1260 01:21:03,530 --> 01:21:07,550 complexity point of view, and also algebraically. 1261 01:21:07,550 --> 01:21:09,840 OK, well that brings us to the end. 1262 01:21:09,840 --> 01:21:13,645 There's actually one more style of graph realization 1263 01:21:13,645 --> 01:21:17,700 that I want to mention to you, for the Reed-Muller codes. 1264 01:21:17,700 --> 01:21:21,210 We know that they're basically based on Hadamard transforms. 1265 01:21:21,210 --> 01:21:25,760 It's a very nice kind of Fourier transform like graph 1266 01:21:25,760 --> 01:21:28,180 realization of Hadamard transforms. 1267 01:21:28,180 --> 01:21:31,360 So that gives us another style of realization of Reed-Muller 1268 01:21:31,360 --> 01:21:35,210 codes that again, we can aggregate to lead to a lot of 1269 01:21:35,210 --> 01:21:39,000 nice structures. 1270 01:21:39,000 --> 01:21:45,160 Again, at the end of the day, unless we allow cycles, we 1271 01:21:45,160 --> 01:21:47,090 don't really gain anything. 1272 01:21:47,090 --> 01:21:51,260 So I'll do that the first part of next time and then do the 1273 01:21:51,260 --> 01:21:52,846 sum-product algorithm. 1274 01:21:52,846 --> 01:21:55,380 I hope I can do it in one lecture.