WEBVTT 00:00:00.090 --> 00:00:02.490 The following content is provided under a Creative 00:00:02.490 --> 00:00:04.030 Commons license. 00:00:04.030 --> 00:00:06.360 Your support will help MIT OpenCourseWare 00:00:06.360 --> 00:00:10.720 continue to offer high quality educational resources for free. 00:00:10.720 --> 00:00:13.320 To make a donation or view additional materials 00:00:13.320 --> 00:00:17.280 from hundreds of MIT courses, visit MIT OpenCourseWare 00:00:17.280 --> 00:00:18.450 at ocw.mit.edu. 00:00:21.364 --> 00:00:23.530 ERIK DEMAINE: All right, today we continue our theme 00:00:23.530 --> 00:00:25.480 of integer data structures. 00:00:25.480 --> 00:00:28.060 And today we're going to look at priority queues instead 00:00:28.060 --> 00:00:30.850 of the predecessor problem, so something a little different. 00:00:30.850 --> 00:00:33.980 Insert, delete, and find min. 00:00:33.980 --> 00:00:38.140 It turns out priority queues are equivalent to sorting 00:00:38.140 --> 00:00:54.022 in the sense that if you have, normally they say f of n, 00:00:54.022 --> 00:00:56.230 but here we're going to depend on both n and the word 00:00:56.230 --> 00:00:59.170 size, priority queue. 00:01:02.830 --> 00:01:05.530 Well then, of course you get a sorting algorithm. 00:01:05.530 --> 00:01:08.420 Just insert everything into the priority queue 00:01:08.420 --> 00:01:11.560 and then find min delete, find min delete, and so on. 00:01:15.350 --> 00:01:20.080 So that's the trivial direction of the equivalence. 00:01:20.080 --> 00:01:23.550 But it turns out, both of these are true. 00:01:23.550 --> 00:01:28.060 If you have a sorting algorithm with n times f time, 00:01:28.060 --> 00:01:30.235 then you can convert it into a priority queue 00:01:30.235 --> 00:01:37.280 that in the worst case every operation takes only f of n, w. 00:01:37.280 --> 00:01:41.550 So that's the equivalence as the result of Thorpe 00:01:41.550 --> 00:01:44.890 from a few years ago, 2007. 00:01:44.890 --> 00:01:46.180 Kind of a cool breakthrough. 00:01:46.180 --> 00:01:48.730 There were earlier versions that weren't quite as efficient 00:01:48.730 --> 00:01:51.700 but this gives you perfect efficiency, turning any sorting 00:01:51.700 --> 00:01:53.050 algorithm into priority queue. 00:01:53.050 --> 00:01:54.610 Rough sketch of the data structure 00:01:54.610 --> 00:01:57.790 is you have a bunch of sorting algorithms which you run now 00:01:57.790 --> 00:02:00.730 and then to get little parts of the input 00:02:00.730 --> 00:02:02.810 in order to build an overall priority queue. 00:02:02.810 --> 00:02:04.990 But it's very complicated. 00:02:04.990 --> 00:02:06.490 There was an entire class project 00:02:06.490 --> 00:02:09.699 on assimilating it and understanding it. 00:02:09.699 --> 00:02:13.010 It's a bit beyond what we can cover in the class. 00:02:13.010 --> 00:02:14.770 So I'm going to take this as given 00:02:14.770 --> 00:02:16.812 and tell you instead about sorting algorithms 00:02:16.812 --> 00:02:19.270 because there's a major open problem here which is, can you 00:02:19.270 --> 00:02:23.470 sort n integers, n w bit integers in a word ram's, 00:02:23.470 --> 00:02:25.750 word size w in linear time. 00:02:25.750 --> 00:02:29.524 We still don't know but we have a bunch of results. 00:02:29.524 --> 00:02:30.940 In particular, we're going to talk 00:02:30.940 --> 00:02:33.460 about a couple interesting cases where 00:02:33.460 --> 00:02:36.070 we can get linear time sorting. 00:02:36.070 --> 00:02:38.830 If you've seen radix sort you know another interesting case 00:02:38.830 --> 00:02:40.495 where linear time sorting is possible. 00:02:44.410 --> 00:02:48.560 Interesting open problem here. 00:02:48.560 --> 00:02:53.620 Can you get constant time decrease key? 00:02:53.620 --> 00:02:56.150 And ideally, also insertion? 00:02:56.150 --> 00:02:58.594 But decrease key would be handy because then this 00:02:58.594 --> 00:02:59.260 would be useful. 00:03:02.530 --> 00:03:04.630 In this reduction if you have a sorting algorithm, 00:03:04.630 --> 00:03:06.550 like to get a priority queue with constant time decrease 00:03:06.550 --> 00:03:07.750 key, that would have applications 00:03:07.750 --> 00:03:08.750 to Dijkstra's algorithm. 00:03:08.750 --> 00:03:10.960 Of course, if you can sort in linear time 00:03:10.960 --> 00:03:13.130 then everything can be done in constant time. 00:03:13.130 --> 00:03:15.430 So this is not the most interesting open problem 00:03:15.430 --> 00:03:19.400 but it would be a step in that direction. 00:03:19.400 --> 00:03:19.900 Cool. 00:03:19.900 --> 00:03:26.365 So let me tell you some bounds about sorting. 00:03:32.482 --> 00:03:38.556 So we have n w bit integers. 00:03:38.556 --> 00:03:40.180 And you take any of these bounds divide 00:03:40.180 --> 00:03:46.250 by n you get the priority queue bound, the best that we know. 00:03:46.250 --> 00:03:52.690 So of course, there's comparison sorting, which takes n log n. 00:03:52.690 --> 00:03:53.950 So that's an upper bound. 00:03:53.950 --> 00:03:55.070 Trying to beat that. 00:03:55.070 --> 00:03:57.610 Trying to get rid of the log. 00:03:57.610 --> 00:04:01.300 There's a counting sort from undergraduate algorithms. 00:04:01.300 --> 00:04:04.320 That takes n plus u time because you need 00:04:04.320 --> 00:04:06.040 to build a table of size u. 00:04:06.040 --> 00:04:08.215 You've got to zero out all the initial counts 00:04:08.215 --> 00:04:11.320 so that requires order u time. 00:04:11.320 --> 00:04:20.470 This is interesting when the word size equals log n. 00:04:20.470 --> 00:04:23.120 So that's a case of potential interest, 00:04:23.120 --> 00:04:24.740 because then 2 to the w is just n. 00:04:24.740 --> 00:04:26.760 So this is linear time. 00:04:26.760 --> 00:04:30.880 But even if w is 2 log n, radix or 1.1 times log n, 00:04:30.880 --> 00:04:33.250 then this is bigger n log n. 00:04:33.250 --> 00:04:34.820 So, not very good. 00:04:34.820 --> 00:04:41.110 But we have radix sort, which is a bit better. 00:04:41.110 --> 00:04:49.310 And this is n times w divided by log n. 00:04:49.310 --> 00:04:53.320 If you do it right you have some word size w, 00:04:53.320 --> 00:04:57.190 you're using counting sort for each digit in the radix. 00:04:57.190 --> 00:05:01.780 So you can afford that the word size of doing counting sort 00:05:01.780 --> 00:05:03.010 to be exactly log n. 00:05:03.010 --> 00:05:05.260 Which means you have your overall word you're dividing 00:05:05.260 --> 00:05:07.210 into chunks of size log n. 00:05:07.210 --> 00:05:09.140 Each of them you can do in linear time. 00:05:09.140 --> 00:05:12.520 So the number of chunks you have to do is w divided by log n. 00:05:12.520 --> 00:05:18.040 And this is linear if w equals order log n. 00:05:22.490 --> 00:05:24.010 Right, but only then. 00:05:24.010 --> 00:05:28.882 So this buys you a little bit, but still not great. 00:05:28.882 --> 00:05:30.590 OK, there's some other sorting algorithms 00:05:30.590 --> 00:05:33.860 we've implicitly seen because we've seen Van Emde Boas. 00:05:33.860 --> 00:05:36.010 I mean, any predecessor data structure 00:05:36.010 --> 00:05:37.760 gives you a priority queue data structure. 00:05:37.760 --> 00:05:42.530 Priority queues-- I didn't write it, but I mentioned-- 00:05:42.530 --> 00:05:45.710 insert, delete, find min. 00:05:45.710 --> 00:05:50.600 Find min is a weaker operation than predecessor or successor. 00:05:50.600 --> 00:05:57.350 So we can take Van Emde Boas's predecessor data structure 00:05:57.350 --> 00:06:01.310 and convert it into a sorting algorithm and we 00:06:01.310 --> 00:06:07.160 get order n times log w. 00:06:07.160 --> 00:06:10.130 Now this is a little bit tricky to compare. 00:06:10.130 --> 00:06:12.560 Of course, log w sounds better than w. 00:06:12.560 --> 00:06:15.110 But there's just divided by log n, which makes 00:06:15.110 --> 00:06:18.620 it a little hard to compare. 00:06:18.620 --> 00:06:25.160 It turns out there's another version. 00:06:25.160 --> 00:06:28.520 You can take this sorting algorithm 00:06:28.520 --> 00:06:33.520 and achieve a bound of m log w over log n. 00:06:36.680 --> 00:06:39.500 OK, so this is a slight improvement in here. 00:06:39.500 --> 00:06:42.200 It's similar to your problem set this week. 00:06:42.200 --> 00:06:43.850 This specific problem was a problem 00:06:43.850 --> 00:06:47.370 set in 2005 of this class. 00:06:47.370 --> 00:06:48.920 So you can check it out if you want 00:06:48.920 --> 00:06:50.900 to see how to do this slight improvement. 00:06:50.900 --> 00:06:52.814 But now this is clearly better than this 00:06:52.814 --> 00:06:54.230 because here we have w over log n. 00:06:54.230 --> 00:06:56.070 Here I've got a log in front. 00:06:56.070 --> 00:06:58.540 So this is a win over radex sort. 00:06:58.540 --> 00:07:01.820 So Van Emde Boas, here's another way in which is interesting. 00:07:01.820 --> 00:07:05.830 It gives us a strict improvement over radex sort. 00:07:05.830 --> 00:07:14.020 And in particular, actually both of these 00:07:14.020 --> 00:07:22.310 are order n log log n if w is not too big, poly log n. 00:07:26.090 --> 00:07:28.130 This doesn't actually matter but it's 00:07:28.130 --> 00:07:30.710 clearly better than this one. 00:07:30.710 --> 00:07:34.490 Again, if w is order log n you get this to be linear. 00:07:34.490 --> 00:07:38.600 But in particular, it says overall you get n log log n up 00:07:38.600 --> 00:07:41.300 to w being poly log. 00:07:41.300 --> 00:07:44.450 OK, why do we care? 00:07:44.450 --> 00:07:46.670 Because there's this other nice algorithm, 00:07:46.670 --> 00:07:51.260 signature sort, which covers all large word sizes. 00:07:51.260 --> 00:07:53.570 And this is an algorithm we will cover today. 00:07:58.370 --> 00:08:01.520 It only works when w is large, and large means 00:08:01.520 --> 00:08:04.850 log to the 2 plus epsilon. 00:08:04.850 --> 00:08:07.740 And this is no matter what epsilon is. 00:08:07.740 --> 00:08:10.250 In reality, it's probably log squared times 00:08:10.250 --> 00:08:12.140 log log n squared. 00:08:12.140 --> 00:08:14.750 But be a little sloppy and just say, 00:08:14.750 --> 00:08:18.410 a little bit bigger than log squared. 00:08:18.410 --> 00:08:20.640 And then it runs in linear time. 00:08:20.640 --> 00:08:25.640 OK, so if you were graphing this in your mind you've got time 00:08:25.640 --> 00:08:28.430 and there's a nice bound here which is order n. 00:08:28.430 --> 00:08:32.960 Then we have everything's above n log n. 00:08:32.960 --> 00:08:35.200 Maybe here we have-- 00:08:35.200 --> 00:08:36.260 is hard to draw-- 00:08:36.260 --> 00:08:39.590 n log log n. 00:08:39.590 --> 00:08:42.630 This is asymptotic space, which doesn't really exist, 00:08:42.630 --> 00:08:44.330 but you know what I mean. 00:08:44.330 --> 00:08:49.475 And then this is the word size versus log n. 00:08:49.475 --> 00:08:50.420 Wow, this is hard to-- 00:08:50.420 --> 00:08:51.800 this is a really weird picture. 00:08:51.800 --> 00:08:53.840 I've never drawn this before. 00:08:53.840 --> 00:08:58.080 So you know, initially in a radex sort, OK log n. 00:08:58.080 --> 00:09:04.125 We could get linear time or order log n, great. 00:09:04.125 --> 00:09:05.750 What this is saying is once you get out 00:09:05.750 --> 00:09:10.520 to log the 2 plus epsilon n you can also do linear. 00:09:10.520 --> 00:09:13.180 And from here over we have linear time. 00:09:13.180 --> 00:09:15.840 And here over we have linear time. 00:09:15.840 --> 00:09:18.635 So the question is what about this middle range between log 00:09:18.635 --> 00:09:22.190 n and log to the 2 plus epsilon? 00:09:22.190 --> 00:09:27.530 Well, one thing we could do is use Van Emde Boas sort, 00:09:27.530 --> 00:09:29.764 get an n log log n bound. 00:09:29.764 --> 00:09:32.180 And so that's one way to fill in this range in the middle. 00:09:32.180 --> 00:09:35.570 And that means overall you can always sort n integers in n log 00:09:35.570 --> 00:09:36.920 log n time. 00:09:36.920 --> 00:09:39.620 If w is big you use signature sort, 00:09:39.620 --> 00:09:42.170 if w small use Van Emde Boas sort. 00:09:42.170 --> 00:09:49.430 So this implies n log log n for all w. 00:09:49.430 --> 00:09:51.080 So that's something. 00:09:51.080 --> 00:09:53.724 It's not the best known. 00:09:53.724 --> 00:10:00.680 Let's see, first result in this direction was by Han in 2001. 00:10:00.680 --> 00:10:10.370 And it achieved this bound, deterministic and in the ac 0 00:10:10.370 --> 00:10:14.390 ram, so without using multiplication. 00:10:14.390 --> 00:10:17.310 Which, I guess Van Emde Boas doesn't either, 00:10:17.310 --> 00:10:19.760 unless I'm missing something. 00:10:19.760 --> 00:10:21.750 But it got rid of the randomisation 00:10:21.750 --> 00:10:25.800 which is necessary to get linear space in Van Emde Boas. 00:10:25.800 --> 00:10:30.258 The current best result on sorting 00:10:30.258 --> 00:10:34.330 is by Han and Thorpe in 2002, so a while ago. 00:10:34.330 --> 00:10:42.150 They achieved n square root log w over log n, 00:10:42.150 --> 00:10:46.560 which is similar to this bound, just 00:10:46.560 --> 00:10:50.200 has an extra square root here. 00:10:50.200 --> 00:10:52.680 So it's a little bit better than Van Emde Boas sort. 00:10:52.680 --> 00:10:53.850 It uses some clever tricks. 00:10:53.850 --> 00:10:54.910 It is randomized. 00:10:58.780 --> 00:11:03.840 So still the best deterministic bound is n log log n, 00:11:03.840 --> 00:11:09.790 And let's see, we can plug that in for this range here. 00:11:09.790 --> 00:11:16.050 And it will imply that you get n square root log 00:11:16.050 --> 00:11:20.130 log n for all w. 00:11:20.130 --> 00:11:26.340 Because in particular, if w is poly log n, 00:11:26.340 --> 00:11:28.560 then this is just log log n. 00:11:28.560 --> 00:11:31.780 So you get this bound. 00:11:31.780 --> 00:11:35.050 Of course, for all w because large w are already covered. 00:11:35.050 --> 00:11:39.176 So this sorting algorithm is a little bit messy, and possibly 00:11:39.176 --> 00:11:40.050 not the right answer. 00:11:40.050 --> 00:11:42.540 I would conjecture you can sort n integers 00:11:42.540 --> 00:11:45.450 in linear time for all w. 00:11:45.450 --> 00:11:46.815 But that is a big open problem. 00:11:52.590 --> 00:11:55.050 This is the current state of the art. 00:11:55.050 --> 00:11:59.300 So this open question is really about for small w 00:11:59.300 --> 00:12:00.870 when w is poly log n, or actually 00:12:00.870 --> 00:12:03.540 when it's between log and poly log. 00:12:03.540 --> 00:12:06.330 So to give you a piece of the complete picture 00:12:06.330 --> 00:12:08.934 I want to tell you about signature sort. 00:12:08.934 --> 00:12:11.100 Because I think you have to use different techniques 00:12:11.100 --> 00:12:12.390 when w is small versus large. 00:12:12.390 --> 00:12:15.540 We know that from fusion trees and Van Emde Boas. 00:12:15.540 --> 00:12:18.480 So I'll tell you about the slightly large case when 00:12:18.480 --> 00:12:21.090 w is at least log to the 2 plus epsilon, 00:12:21.090 --> 00:12:22.600 how to get linear time. 00:12:22.600 --> 00:12:23.710 And this is fun. 00:12:23.710 --> 00:12:25.950 It uses bit tricks, it uses a lot of cool ideas, 00:12:25.950 --> 00:12:28.470 uses hashing, lots of things. 00:12:28.470 --> 00:12:32.820 It also uses another sorting algorithm called packed sort. 00:12:32.820 --> 00:12:35.310 This doesn't fit in the regular table 00:12:35.310 --> 00:12:38.700 because it's for a situation when more than one input 00:12:38.700 --> 00:12:40.780 number fits in a word. 00:12:40.780 --> 00:12:43.290 So we're going to let b denote the number of bits 00:12:43.290 --> 00:12:44.340 in the inputs. 00:12:44.340 --> 00:12:46.560 So we have b bit integers. 00:12:46.560 --> 00:12:50.460 Then when w is at least b times log n log log n, 00:12:50.460 --> 00:12:54.540 so you can fit log n log log n integers in a single word, 00:12:54.540 --> 00:12:56.982 then you can sort them in linear time. 00:12:56.982 --> 00:12:59.190 So it's like your machine is just a little bit bigger 00:12:59.190 --> 00:13:01.850 than what you'd need to represent the integers, 00:13:01.850 --> 00:13:04.890 then all sorting can be done in a linear time. 00:13:04.890 --> 00:13:08.580 And as you might imagine, this log to the 2 plus epsilon 00:13:08.580 --> 00:13:12.660 buys you a little bit of room with some tricks, 00:13:12.660 --> 00:13:15.350 namely hashing. 00:13:15.350 --> 00:13:19.140 Right, that's where we're going. 00:13:19.140 --> 00:13:21.187 And we'll use yet another sorting algorithm, 00:13:21.187 --> 00:13:22.770 bitonic sort, which you may have seen. 00:13:22.770 --> 00:13:25.840 It's a sorting network. 00:13:25.840 --> 00:13:27.840 It's in CLRS up to second edition, 00:13:27.840 --> 00:13:29.617 but it got removed from third edition. 00:13:29.617 --> 00:13:31.950 I guess they thought people didn't use sorting networks, 00:13:31.950 --> 00:13:33.360 but we use sorting networks here. 00:13:36.270 --> 00:13:39.690 Because it gives us parallel sorting and the word ram 00:13:39.690 --> 00:13:43.290 is all about using the parallelism in your word. 00:13:43.290 --> 00:13:48.420 So, we'll get to those. 00:13:48.420 --> 00:13:53.300 I'm going to start at the top level, which is signature sort. 00:13:53.300 --> 00:13:55.050 And we're going to leave things to be done 00:13:55.050 --> 00:13:57.420 and then pack sorting will be one of them. 00:13:57.420 --> 00:13:59.670 And bitonic sorting will be used in pack sorting. 00:13:59.670 --> 00:14:03.830 So it's a chain like this. 00:14:03.830 --> 00:14:05.580 Cool. 00:14:05.580 --> 00:14:10.770 So I'm going to assume my signature 00:14:10.770 --> 00:14:15.990 sort that w is at least log to 2 pulse epsilon n 00:14:15.990 --> 00:14:18.640 times log log n. 00:14:18.640 --> 00:14:23.407 This is really just a change in what epsilon means. 00:14:23.407 --> 00:14:25.240 If you're given some particular epsilon make 00:14:25.240 --> 00:14:26.906 it a little smaller, then you can afford 00:14:26.906 --> 00:14:28.560 to add that log log factor. 00:14:28.560 --> 00:14:32.260 This is just for convenience. 00:14:32.260 --> 00:14:37.927 Now here's the idea. 00:14:37.927 --> 00:14:39.510 It's a pretty cool idea I have to say. 00:14:39.510 --> 00:14:59.250 It's by Anderson and others in 1998. 00:14:59.250 --> 00:15:01.896 OK, word size is big, which is both good and bad. 00:15:01.896 --> 00:15:03.770 It means the numbers we have to sort are big. 00:15:03.770 --> 00:15:07.600 They have a lot of bits in them, way more than log n bits. 00:15:07.600 --> 00:15:10.090 But also, we have this powerful machine 00:15:10.090 --> 00:15:13.794 that can handle words of very large size 00:15:13.794 --> 00:15:15.460 and do things and constant time on them. 00:15:19.090 --> 00:15:21.160 I'd like to make some slop. 00:15:21.160 --> 00:15:23.320 I want my words to be bigger than my items, 00:15:23.320 --> 00:15:26.410 so first few steps are about doing that. 00:15:26.410 --> 00:15:30.340 The first idea is, well, you've got this giant word 00:15:30.340 --> 00:15:32.020 log to the 2 plus epsilon. 00:15:32.020 --> 00:15:34.840 So let's split it into chunks. 00:15:34.840 --> 00:15:40.180 And the number of chunks is log to the epsilon. 00:15:40.180 --> 00:15:44.200 So that means each chunk is-- 00:15:44.200 --> 00:15:45.710 well, it could be even bigger. 00:15:45.710 --> 00:15:48.020 So it's at least log squared n. 00:15:48.020 --> 00:15:51.510 But it could be bigger than that because w 00:15:51.510 --> 00:15:53.572 is at least log to the 2 plus epsilon. 00:15:53.572 --> 00:15:54.280 So we don't know. 00:15:54.280 --> 00:15:57.330 These things are of size w divided by log to the epsilon 00:15:57.330 --> 00:15:57.830 n. 00:15:57.830 --> 00:16:00.470 So it's at least log squared. 00:16:00.470 --> 00:16:04.970 OK, here's the crazy idea. 00:16:04.970 --> 00:16:07.480 We're thinking of our number being divided 00:16:07.480 --> 00:16:10.180 into these chunks, you know, it's being represented 00:16:10.180 --> 00:16:15.070 in some base, 2 to the w divided by log to the epsilon n. 00:16:15.070 --> 00:16:17.510 It's a bit of a mouthful. 00:16:17.510 --> 00:16:20.840 But think about these digits. 00:16:20.840 --> 00:16:27.040 I claim if you look at a digit, most values don't appear. 00:16:27.040 --> 00:16:29.920 Because how many values are there for a digit? 00:16:29.920 --> 00:16:33.310 Well, it's at least 2 to the log squared n. 00:16:33.310 --> 00:16:34.220 That's a big number. 00:16:34.220 --> 00:16:36.610 That's way bigger than polynomial and n. 00:16:36.610 --> 00:16:39.790 But how many different digits are there? 00:16:39.790 --> 00:16:41.860 Well, there's n numbers, each of them 00:16:41.860 --> 00:16:47.350 has log to the epsilon n digits in it appearing. 00:16:47.350 --> 00:16:52.870 And so there's roughly a linear number of digit values total. 00:16:52.870 --> 00:16:54.850 But the number of possible digit values 00:16:54.850 --> 00:16:57.010 is 2 to the log log squared n, which 00:16:57.010 --> 00:17:00.580 is huge, super polynomial. 00:17:00.580 --> 00:17:05.030 So most digit values don't appear in our numbers. 00:17:05.030 --> 00:17:07.359 So what we're going to do is hash the digit 00:17:07.359 --> 00:17:09.177 values to reduce their space. 00:17:09.177 --> 00:17:11.760 Because they're currently living in a universe of 2 to the log 00:17:11.760 --> 00:17:14.444 squared n, but there's only n times log 00:17:14.444 --> 00:17:16.089 to the epsilon n of them. 00:17:16.089 --> 00:17:23.200 So we should be able to reduce to have a much smaller universe 00:17:23.200 --> 00:17:24.970 using hashing. 00:17:24.970 --> 00:17:30.669 Maybe perfect hashing even. 00:17:30.669 --> 00:17:32.710 I mean, we don't have to be too intelligent here. 00:17:32.710 --> 00:17:33.585 Tell you what we get. 00:17:41.980 --> 00:17:46.120 What I'd like is to reduce to a polynomial universe. 00:17:46.120 --> 00:17:53.380 And another way to say that is that the hash values are order 00:17:53.380 --> 00:17:54.240 log n bits. 00:17:54.240 --> 00:17:57.140 So this lets us go to n to the 10, or whatever, 00:17:57.140 --> 00:17:58.360 which is plenty of room. 00:17:58.360 --> 00:18:03.050 We take any universal hash function, 00:18:03.050 --> 00:18:06.250 as soon as you get to roughly n squared size, 00:18:06.250 --> 00:18:08.380 I mean the square of the number of things, 00:18:08.380 --> 00:18:10.750 by a birthday paradox, constant probability none of them 00:18:10.750 --> 00:18:12.070 will hit each other. 00:18:12.070 --> 00:18:14.320 That was the second level of perfect hashing. 00:18:14.320 --> 00:18:17.140 We don't need two levels here, just say OK, user order 00:18:17.140 --> 00:18:17.760 log n bits. 00:18:17.760 --> 00:18:20.320 You can set this to 100, whatever. 00:18:20.320 --> 00:18:23.230 Then with high probability you will not get collisions, 00:18:23.230 --> 00:18:25.300 so none of the chunks will hit each other. 00:18:25.300 --> 00:18:27.880 So I'm omitting a step here which is how to-- 00:18:27.880 --> 00:18:31.370 OK, I'm omitting a couple of things here. 00:18:31.370 --> 00:18:36.130 One of them is I want to do this in linear time. 00:18:36.130 --> 00:18:37.967 Everything has to be in linear time. 00:18:37.967 --> 00:18:39.550 So one front part is I have to compute 00:18:39.550 --> 00:18:43.210 the hash value of all of these digits 00:18:43.210 --> 00:18:47.220 in a single word in constant time. 00:18:47.220 --> 00:18:51.550 So a brief aside, how would I compute a universal hash 00:18:51.550 --> 00:18:55.240 function of these digits individually in constant time? 00:18:55.240 --> 00:19:02.230 Well, my personal favorite universal hashing scheme 00:19:02.230 --> 00:19:03.940 is the multiplication method. 00:19:03.940 --> 00:19:08.240 You just need to multiply by a single value m 00:19:08.240 --> 00:19:11.210 and then take it modular something. 00:19:11.210 --> 00:19:20.690 So if I do multiplication, I mean 00:19:20.690 --> 00:19:22.490 if you think of there being something here, 00:19:22.490 --> 00:19:25.475 something here, something here, something here, something here, 00:19:25.475 --> 00:19:30.481 m does multiply each of these individually. 00:19:34.170 --> 00:19:36.200 Wow, that's a rough sketch. 00:19:36.200 --> 00:19:38.240 I want to hash each of these guys multiply, 00:19:38.240 --> 00:19:40.220 them by the multiplication method. 00:19:40.220 --> 00:19:42.800 This would work fine, except you get overflows, 00:19:42.800 --> 00:19:44.340 which is a bit messy. 00:19:44.340 --> 00:19:47.210 So in fact, I'm going to have to do it for the odd guys 00:19:47.210 --> 00:19:50.390 first, multiply them out, mask out the bits I care about, 00:19:50.390 --> 00:19:53.600 then do it for the even guys separately. 00:19:53.600 --> 00:19:54.470 So it's a bit messy. 00:19:54.470 --> 00:19:58.700 But hand-wavy proof you can compute the hash function 00:19:58.700 --> 00:20:00.230 of each of these guys individually, 00:20:00.230 --> 00:20:02.100 but in constant time. 00:20:02.100 --> 00:20:03.260 So that lets me do this. 00:20:03.260 --> 00:20:04.640 You have to then check for collisions, 00:20:04.640 --> 00:20:05.723 which is kind of annoying. 00:20:05.723 --> 00:20:06.900 Let's not worry about that. 00:20:06.900 --> 00:20:08.750 Set this really high, very high probability, 00:20:08.750 --> 00:20:11.900 there will be no collisions. 00:20:11.900 --> 00:20:13.640 Cool. 00:20:13.640 --> 00:20:17.319 So at this point we have mucked up all the digits. 00:20:17.319 --> 00:20:18.860 The annoying thing about this hashing 00:20:18.860 --> 00:20:20.450 is we do not preserve order. 00:20:24.860 --> 00:20:26.840 It's a little weird for a sorting algorithm not 00:20:26.840 --> 00:20:28.670 preserve order. 00:20:28.670 --> 00:20:31.850 The whole point is to put things in order. 00:20:31.850 --> 00:20:34.370 But it's going to be useful nonetheless. 00:20:34.370 --> 00:20:36.740 And so each digit has been mangled. 00:20:36.740 --> 00:20:38.570 Each digit is out of order. 00:20:38.570 --> 00:20:40.910 But of course, the sequence of digits 00:20:40.910 --> 00:20:45.800 has been preserved, in some sense 00:20:45.800 --> 00:20:47.730 So, what? 00:20:47.730 --> 00:20:50.300 Well, at this point we can use packed 00:20:50.300 --> 00:20:52.640 sorting to sort what remains. 00:21:05.836 --> 00:21:08.670 In general packed sorting can sort n 00:21:08.670 --> 00:21:13.680 b bit integers provided b is small, so w 00:21:13.680 --> 00:21:18.500 is at least b log n log log n. 00:21:18.500 --> 00:21:21.900 That's the theorem of packed sorting. 00:21:21.900 --> 00:21:25.080 We're going to delay the coverage of this result 00:21:25.080 --> 00:21:26.330 until later. 00:21:26.330 --> 00:21:36.830 And I claim that that applies here because each chunk is now 00:21:36.830 --> 00:21:39.290 order log n bits. 00:21:39.290 --> 00:21:43.200 And there's log to the epsilon n bits. 00:21:43.200 --> 00:21:51.320 So b is theta log to the 1 plus epsilon n. 00:21:51.320 --> 00:22:01.890 And w is at least log to the 2 plus epsilon times log log n. 00:22:01.890 --> 00:22:04.670 OK, so the ratio between these is as needed. 00:22:04.670 --> 00:22:06.844 It's at least log n times log log n. 00:22:09.450 --> 00:22:11.220 OK, so that's why this is good. 00:22:11.220 --> 00:22:13.250 There were these things were huge before. 00:22:13.250 --> 00:22:15.330 We've compressed them a lot. 00:22:15.330 --> 00:22:18.560 They're at least log square, now they're only log. 00:22:18.560 --> 00:22:21.265 And so now we can apply packed sorting. 00:22:24.080 --> 00:22:28.460 You can fit log times log log items into one word. 00:22:28.460 --> 00:22:30.680 And that's going to make sorting easy, somehow. 00:22:30.680 --> 00:22:32.660 That's to be done. 00:22:32.660 --> 00:22:36.800 But now the question is, suppose we had packed sorting, 00:22:36.800 --> 00:22:37.760 how does this help you? 00:22:37.760 --> 00:22:40.460 Because this does not sort the original integers. 00:22:40.460 --> 00:22:42.290 It sorts these integers that have 00:22:42.290 --> 00:22:45.140 all of their digits permuted in some random way. 00:22:45.140 --> 00:22:49.070 We have no idea how the hash function re-orders things. 00:22:49.070 --> 00:22:51.977 So we're not done with signature sort. 00:22:51.977 --> 00:22:54.560 It's called signature sort, of course, because of the hashing. 00:22:54.560 --> 00:22:57.390 You're taking signatures of each of these digits. 00:22:57.390 --> 00:22:59.690 Another word for hashes. 00:22:59.690 --> 00:23:05.955 All right, next step is to build a compressed trie. 00:23:09.380 --> 00:23:11.060 This is a term we haven't used yet. 00:23:17.005 --> 00:23:18.380 So I'm going to call these things 00:23:18.380 --> 00:23:21.890 that we sorted signatures. 00:23:21.890 --> 00:23:26.960 And how many people know what a compressed trie is? 00:23:26.960 --> 00:23:27.820 OK, no one. 00:23:27.820 --> 00:23:28.869 Wow, just a few. 00:23:28.869 --> 00:23:30.410 So you haven't seen suffix trees yet. 00:23:30.410 --> 00:23:32.930 We're going to cover where compressed tries come 00:23:32.930 --> 00:23:34.490 from in a couple of lectures but we 00:23:34.490 --> 00:23:38.480 get to see them a little early because they're cool and needed 00:23:38.480 --> 00:23:41.640 in this algorithm. 00:23:41.640 --> 00:23:44.230 So first notion is the notion of a trie. 00:23:44.230 --> 00:23:49.340 The word trie comes from retrieval, 00:23:49.340 --> 00:23:51.870 from old days when information retrieval was 00:23:51.870 --> 00:23:56.640 what's now called web searching, or whatever. 00:23:56.640 --> 00:24:00.410 So it's now pronounced trie. 00:24:00.410 --> 00:24:02.630 It's just another word for tree, but where 00:24:02.630 --> 00:24:06.530 the children of the tree come from a fixed universe. 00:24:10.780 --> 00:24:13.285 We've been working with tries all the time 00:24:13.285 --> 00:24:16.990 in pretty much every lecture for the last n lectures, 00:24:16.990 --> 00:24:19.620 last w lectures, I don't know, a lot of them. 00:24:19.620 --> 00:24:21.790 We said oh, OK, well let's just consider 00:24:21.790 --> 00:24:24.261 the space of all possible keys. 00:24:24.261 --> 00:24:26.260 You know, I've drawn this picture zillion times. 00:24:26.260 --> 00:24:28.630 Even when we weren't doing Van Emde Boas, which actually 00:24:28.630 --> 00:24:30.970 stores all of these things, we were 00:24:30.970 --> 00:24:32.740 thinking about fusion trees like, oh yeah, 00:24:32.740 --> 00:24:34.690 so we're storing this value, and we're 00:24:34.690 --> 00:24:37.660 storing this value, and maybe this value. 00:24:37.660 --> 00:24:40.080 And we understand this corresponds to a 0-bit. 00:24:40.080 --> 00:24:41.250 These are 0 and 1 bits. 00:24:41.250 --> 00:24:43.560 This is a 1, 0, 1, whatever. 00:24:43.560 --> 00:24:47.030 So we understand how to convert a bit string into this picture. 00:24:47.030 --> 00:24:49.750 This is what we call a binary trie. 00:24:49.750 --> 00:24:51.560 I want to do the same thing over here, 00:24:51.560 --> 00:24:53.350 except it's not a binary trie. 00:24:53.350 --> 00:24:56.320 Now each digit is giant so we're going to have 00:24:56.320 --> 00:24:58.570 some huge branching factor. 00:24:58.570 --> 00:25:00.610 So it's going to be more like a b-tree. 00:25:00.610 --> 00:25:03.280 The first node has a whole bunch of children. 00:25:03.280 --> 00:25:04.540 I'm not going to even-- 00:25:04.540 --> 00:25:11.060 I mean, it's 2 to the w divided by log to the epsilon n. 00:25:11.060 --> 00:25:13.440 That's the branching factor. 00:25:13.440 --> 00:25:15.430 OK, it's big. 00:25:15.430 --> 00:25:19.570 That's the number of possible values in the first digit. 00:25:19.570 --> 00:25:20.860 OK. 00:25:20.860 --> 00:25:27.190 So then maybe this child has some node there. 00:25:27.190 --> 00:25:29.410 And maybe this one. 00:25:34.130 --> 00:25:37.205 Actually, sorry, I didn't want to draw it that way. 00:25:37.205 --> 00:25:40.450 I have a specific trie in mind, or part of it. 00:25:49.200 --> 00:25:52.545 OK, I mean, trie is going to look something like this. 00:25:52.545 --> 00:25:55.780 It's harder to draw, but it's the larger than binary 00:25:55.780 --> 00:25:57.570 equivalent of this picture. 00:25:57.570 --> 00:26:01.370 This would be a trie. 00:26:01.370 --> 00:26:03.730 So let's think about a particular situation. 00:26:03.730 --> 00:26:08.780 Which is, suppose we just have two items in this structure. 00:26:08.780 --> 00:26:10.310 Then it's easier to draw. 00:26:10.310 --> 00:26:14.350 So let's suppose there's some item over here, x1. 00:26:14.350 --> 00:26:16.510 There's some item over here, x2. 00:26:16.510 --> 00:26:19.402 Now, generic picture is they share some common prefix. 00:26:19.402 --> 00:26:20.485 This is the common prefix. 00:26:24.110 --> 00:26:26.425 It's just to understand some particular picture here. 00:26:26.425 --> 00:26:29.050 Then they diverge, and you know, then they have their own digit 00:26:29.050 --> 00:26:30.034 values. 00:26:30.034 --> 00:26:32.200 In some sense, this is the digit value we care about 00:26:32.200 --> 00:26:34.450 because it distinguishes them. 00:26:34.450 --> 00:26:39.850 So there's this idea of a compressed trie, which cleans 00:26:39.850 --> 00:26:41.170 up this picture a little bit. 00:26:44.772 --> 00:26:46.730 The idea with the compress trie is very simple. 00:26:46.730 --> 00:26:48.050 You draw this trie. 00:26:48.050 --> 00:26:53.350 So here, imagine I'm erasing all the things that do not appear. 00:26:53.350 --> 00:26:55.220 Maybe all this is gone. 00:26:55.220 --> 00:26:57.530 So I'm just keeping this part of the structure. 00:26:57.530 --> 00:27:00.140 Let's do it for the binary case first. 00:27:00.140 --> 00:27:02.790 The idea in a compress trie is very simple. 00:27:02.790 --> 00:27:10.619 I'm going to contract every non-branching node 00:27:10.619 --> 00:27:11.285 into its parent. 00:27:21.930 --> 00:27:23.340 So what that means. 00:27:23.340 --> 00:27:28.950 So here I'm storing three values: 0, 0, 1, 1; 0, 1, 00:27:28.950 --> 00:27:34.370 0, 0; and 0, 1, 1, 1. 00:27:34.370 --> 00:27:36.680 So some nodes are non-branching, that's 00:27:36.680 --> 00:27:39.605 this node, this node, and this node. 00:27:39.605 --> 00:27:41.480 And the point is well, why store those nodes? 00:27:41.480 --> 00:27:43.938 I mean, there was nothing being distinguished at that node, 00:27:43.938 --> 00:27:45.380 so contract it. 00:27:45.380 --> 00:27:52.510 So the new picture if I redrew this would be 0, 0, 1-- 00:27:52.510 --> 00:27:55.010 sorry, also this is non-branching. 00:27:55.010 --> 00:27:58.537 So in fact, I'm going to contract this 00:27:58.537 --> 00:28:00.120 into its parent, this into its parent. 00:28:00.120 --> 00:28:01.950 So this whole path, this non-branching path 00:28:01.950 --> 00:28:06.140 will be left with a single edge which I will label 0, 1, 1. 00:28:06.140 --> 00:28:08.570 But it's a single edge in this structure. 00:28:08.570 --> 00:28:14.090 Over here we have an edge 1, and now we have 0, 0 here, 00:28:14.090 --> 00:28:16.640 and 1, 1 here. 00:28:16.640 --> 00:28:18.650 That is the compressed trie representation 00:28:18.650 --> 00:28:20.660 of these three values. 00:28:20.660 --> 00:28:22.610 The nice thing about this structure 00:28:22.610 --> 00:28:25.580 is every node is branching except the leaves, which 00:28:25.580 --> 00:28:27.050 means the number of nodes here is 00:28:27.050 --> 00:28:29.230 linear in the number of leaves. 00:28:29.230 --> 00:28:32.960 That's like twice the number of leaves minus 1, roughly. 00:28:32.960 --> 00:28:35.930 I guess the root might not be branching. 00:28:35.930 --> 00:28:37.510 It has no parent to contract into. 00:28:37.510 --> 00:28:38.660 Whatever. 00:28:38.660 --> 00:28:41.612 But more and more nodes, most 2n nodes. 00:28:41.612 --> 00:28:42.570 That's the binary case. 00:28:42.570 --> 00:28:47.010 Now over here what we do is in the corresponding contracted 00:28:47.010 --> 00:28:49.180 try a picture of this thing. 00:28:49.180 --> 00:28:50.915 Let's see, it's a second edge. 00:28:50.915 --> 00:28:53.990 I'm going to put just a label of 2 on this edge to say, well, 00:28:53.990 --> 00:28:54.950 I'm skipping this node. 00:28:54.950 --> 00:28:56.210 This guy is non-branching. 00:28:56.210 --> 00:28:59.840 So I want to get rid of it and contract it into the parent. 00:28:59.840 --> 00:29:02.510 So now this corresponds to two edges here. 00:29:02.510 --> 00:29:04.820 There are two digits, which I could write the digits 00:29:04.820 --> 00:29:05.690 if I knew what they were. 00:29:05.690 --> 00:29:07.731 But particularly I want to keep track of the fact 00:29:07.731 --> 00:29:11.100 that there exactly two of them. 00:29:11.100 --> 00:29:12.740 And then there's a branching node. 00:29:12.740 --> 00:29:14.450 That one stays. 00:29:14.450 --> 00:29:17.040 But then I actually we'll go straight to the leaves here. 00:29:17.040 --> 00:29:22.360 This will be x1 and x2. 00:29:22.360 --> 00:29:24.265 And this will have a lot of digits. 00:29:26.960 --> 00:29:29.000 OK, so some large number written here. 00:29:29.000 --> 00:29:31.810 So that's the idea with a compress trie. 00:29:31.810 --> 00:29:34.030 Now the nice thing about a compress trie 00:29:34.030 --> 00:29:40.000 is they have linear size, number of edges and vertices 00:29:40.000 --> 00:29:42.220 is linear in the number of leaves, 00:29:42.220 --> 00:29:45.760 which is the number of values we have here. 00:29:45.760 --> 00:29:47.440 Another nice thing about compress tries 00:29:47.440 --> 00:29:49.060 is they preserve order. 00:29:49.060 --> 00:29:50.830 So if you look at the order of the leaves 00:29:50.830 --> 00:29:54.287 down here it's the same as the order of the leaves over here. 00:29:54.287 --> 00:29:55.870 We're not really changing order, we're 00:29:55.870 --> 00:29:59.620 just making things a little shorter to write down. 00:29:59.620 --> 00:30:00.820 Same thing over here. 00:30:00.820 --> 00:30:02.500 I mean, x1 appears the left of x2. 00:30:18.130 --> 00:30:20.992 Say the order of the leaves is preserved. 00:30:20.992 --> 00:30:22.700 The other cool thing about compress tries 00:30:22.700 --> 00:30:25.120 is you can compute them in linear time. 00:30:25.120 --> 00:30:28.360 But let me tell you why we're drawing this picture. 00:30:28.360 --> 00:30:32.320 We've messed up all these chunks, all these digits. 00:30:32.320 --> 00:30:34.730 What does that correspond to in this picture? 00:30:34.730 --> 00:30:37.661 We've permuted all of the children in this node. 00:30:37.661 --> 00:30:39.910 We've permuted all the children in this node, permuted 00:30:39.910 --> 00:30:41.320 all the children in this node. 00:30:41.320 --> 00:30:44.020 But otherwise, the structure of the tree is correct. 00:30:44.020 --> 00:30:46.000 What we wanted to do was compute-- 00:30:46.000 --> 00:30:48.610 if we could compute the compress trie representation 00:30:48.610 --> 00:30:53.020 of the original values just having broken the integers 00:30:53.020 --> 00:30:55.919 into these pieces, then if we had that trie we could just 00:30:55.919 --> 00:30:58.210 do an in order traversal, find the order of the leaves, 00:30:58.210 --> 00:31:01.720 then we'd have the sorted order of the integers. 00:31:01.720 --> 00:31:05.590 Instead what we've computed is we didn't compute 00:31:05.590 --> 00:31:06.850 the correct compress trie. 00:31:06.850 --> 00:31:08.350 But we computed a different compress 00:31:08.350 --> 00:31:12.220 trie where every node has its children permuted. 00:31:12.220 --> 00:31:15.340 So all we need to do is put the children within each node 00:31:15.340 --> 00:31:17.860 in the correct order, then we would 00:31:17.860 --> 00:31:20.220 have the correct compress trie. 00:31:20.220 --> 00:31:22.490 OK, that's why we're doing this. 00:31:22.490 --> 00:31:27.400 But first, how do we compute a compress trie in linear time? 00:31:32.080 --> 00:31:33.950 This is a simple amortization argument. 00:31:48.887 --> 00:31:49.970 Remember what we're given. 00:31:53.240 --> 00:31:57.070 At this point we are given the leaves in sorted order. 00:31:57.070 --> 00:31:59.870 We've just sorted our things so all I'm trying to do 00:31:59.870 --> 00:32:02.970 is compute the trie of the stuff above the things. 00:32:02.970 --> 00:32:06.406 OK, we're given the leaves of the trie in the correct order. 00:32:06.406 --> 00:32:08.030 I just need to build the compress trie. 00:32:08.030 --> 00:32:09.260 The reason I'm building a compress trie 00:32:09.260 --> 00:32:10.635 is so I can do it in linear time. 00:32:10.635 --> 00:32:15.780 Constructing the uncompressed trie would take too long. 00:32:15.780 --> 00:32:17.690 OK, so how do we do this? 00:32:26.150 --> 00:32:29.030 We have our items xn in order. 00:32:34.340 --> 00:32:41.570 I want to just insert it into the compress trie. 00:32:41.570 --> 00:32:45.080 So I start with an empty compress trie. 00:32:45.080 --> 00:32:47.840 First thing I do is add x1, so it's just a single leaf. 00:32:47.840 --> 00:32:50.210 Then I add x2 to the trie, x3 to the trie. 00:32:50.210 --> 00:32:53.030 In general, I'll have computed-- 00:32:53.030 --> 00:32:54.760 I'm going to draw a picture. 00:32:54.760 --> 00:32:58.060 How to draw a partial trie, something like this. 00:32:58.060 --> 00:33:00.240 I've computed the left part of trie already. 00:33:00.240 --> 00:33:03.080 All these x1's have been done. 00:33:03.080 --> 00:33:05.780 And now I need to add on the next, 00:33:05.780 --> 00:33:09.930 say xi over here somewhere. 00:33:09.930 --> 00:33:12.860 So think of how that would look in the trie. 00:33:12.860 --> 00:33:15.715 It has to come off of the right side, 00:33:15.715 --> 00:33:17.090 I guess what would you call this? 00:33:17.090 --> 00:33:19.550 The right spine of the tree. 00:33:19.550 --> 00:33:22.310 Some right most node over here. 00:33:22.310 --> 00:33:27.140 We want xi to be a new to the right child. 00:33:27.140 --> 00:33:30.890 So one possible picture is there was 00:33:30.890 --> 00:33:33.830 some node which had some other children, 00:33:33.830 --> 00:33:37.100 maybe this one had xi minus 1. 00:33:37.100 --> 00:33:39.160 Maybe xi belongs over here. 00:33:39.160 --> 00:33:43.700 Maybe this leads to xi in the uncompressed trie. 00:33:43.700 --> 00:33:46.110 This could be a potential picture. 00:33:46.110 --> 00:33:49.310 In the compress trie this node either exists 00:33:49.310 --> 00:33:50.960 or got contracted. 00:33:50.960 --> 00:33:55.250 If it exists maybe there was a previous xi minus 2 00:33:55.250 --> 00:33:56.464 belongs over here. 00:33:56.464 --> 00:33:58.880 So we already had this node because it distinguished these 00:33:58.880 --> 00:34:01.410 two xj's. 00:34:01.410 --> 00:34:03.000 To add the xi is really easy. 00:34:03.000 --> 00:34:07.820 I just add a node here and then which points to a leaf of xi. 00:34:07.820 --> 00:34:10.760 So that could be one situation. 00:34:10.760 --> 00:34:13.480 The other situation is only slightly more complicated. 00:34:21.980 --> 00:34:25.550 So the other possibility is this node 00:34:25.550 --> 00:34:28.730 didn't exist in the compressed trie, which means we jumped 00:34:28.730 --> 00:34:32.909 over it in the compress trie. 00:34:32.909 --> 00:34:35.210 Some number bigger than 1 here, maybe you 00:34:35.210 --> 00:34:39.120 skipped three nodes, who knows. 00:34:39.120 --> 00:34:40.451 So now this node matters. 00:34:40.451 --> 00:34:41.450 It didn't matter before. 00:34:41.450 --> 00:34:44.570 Before it was non-branching, but now we add xi, it's branching. 00:34:44.570 --> 00:34:48.909 All we do is update, add this new guy, 00:34:48.909 --> 00:34:52.389 add the new pointer to xi. 00:34:52.389 --> 00:34:54.949 OK, so constant work. 00:34:54.949 --> 00:34:59.780 Great, constant work per xi inserted, linear time overall. 00:34:59.780 --> 00:35:00.920 Not really. 00:35:00.920 --> 00:35:02.510 I've cheated a little bit here, which 00:35:02.510 --> 00:35:05.780 is how do we know what we're going to do this operation. 00:35:05.780 --> 00:35:08.930 So the changes we make is constant, that's fine. 00:35:08.930 --> 00:35:11.120 But how did I find that this was the edge where 00:35:11.120 --> 00:35:13.100 I need to put xi in? 00:35:13.100 --> 00:35:14.990 You can't do it in constant time, 00:35:14.990 --> 00:35:17.840 but we can do it in constant amortized time. 00:35:17.840 --> 00:35:22.700 So what we're going to do is start at the right most-- 00:35:22.700 --> 00:35:26.690 that is not very red. 00:35:26.690 --> 00:35:29.965 Not a red chalk. 00:35:29.965 --> 00:35:32.210 Red chalk. 00:35:32.210 --> 00:35:35.630 We start at this leaf, the rightmost leaf which you go 00:35:35.630 --> 00:35:37.380 going right most as possible. 00:35:37.380 --> 00:35:42.090 We're going to walk up the tree until we find the right point. 00:35:42.090 --> 00:35:45.340 We're going to spend linear time this path walk. 00:35:45.340 --> 00:35:47.532 OK, how could this possibly be OK? 00:35:47.532 --> 00:35:49.990 I think you believe that I can do it in linear time, right. 00:35:49.990 --> 00:35:51.870 Each point I see-- 00:35:51.870 --> 00:35:53.660 what is my common prefix? 00:35:53.660 --> 00:35:55.470 Actually an easy way is right up front. 00:35:55.470 --> 00:35:59.510 This thing right here is xi minus 1. 00:35:59.510 --> 00:36:00.830 These are single integers. 00:36:00.830 --> 00:36:04.070 So I can just compute their x or, find 00:36:04.070 --> 00:36:06.560 the most significant one bit, which we did with the fusion 00:36:06.560 --> 00:36:08.180 tree lecture. 00:36:08.180 --> 00:36:12.890 And boom, I know where they differ in bit position. 00:36:12.890 --> 00:36:14.630 By some rounding I can figure out 00:36:14.630 --> 00:36:19.070 which chunk they differ in, just by dividing, 00:36:19.070 --> 00:36:19.880 taking the ceiling. 00:36:22.950 --> 00:36:26.180 So I know the correct depth. 00:36:26.180 --> 00:36:30.170 And then these numbers tell me how much depth I'm losing. 00:36:30.170 --> 00:36:32.480 As I walk up at some point I'll discover, 00:36:32.480 --> 00:36:34.160 oops, I just jumped over the depth 00:36:34.160 --> 00:36:36.120 I care about where they differ. 00:36:36.120 --> 00:36:38.480 And so at that point I know to do that operation. 00:36:38.480 --> 00:36:43.280 So if there are k edges here on the compress trie, 00:36:43.280 --> 00:36:48.390 I can do this in order k time, k plus 1, let's say. 00:36:48.390 --> 00:36:50.090 Why is that OK? 00:36:50.090 --> 00:36:55.190 Because my potential function, amortization 00:36:55.190 --> 00:37:03.045 is going to be the length of the rightmost path. 00:37:06.030 --> 00:37:10.020 OK, before my rightmost path was this thing. 00:37:10.020 --> 00:37:13.350 Afterwards, my rightmost path is going to be this thing. 00:37:13.350 --> 00:37:17.270 So it got roughly k shorter, k minus 1 shorter. 00:37:17.270 --> 00:37:19.710 OK, so I charged this k plus 1 cost 00:37:19.710 --> 00:37:24.850 to a potential decrease of k minus 1. 00:37:24.850 --> 00:37:28.250 And so the amortize cost is only constant. 00:37:28.250 --> 00:37:29.670 Easy amortization. 00:37:29.670 --> 00:37:31.620 Another way to see this is, overall 00:37:31.620 --> 00:37:34.050 we're basically doing it in order traversal of the tree. 00:37:34.050 --> 00:37:37.080 So whenever we walk up we'll never visit these nodes again, 00:37:37.080 --> 00:37:39.112 because then we go down at a different place. 00:37:39.112 --> 00:37:41.070 OK, yeah, this guy we're going to walk up again 00:37:41.070 --> 00:37:43.195 because in an in order traversal we would have gone 00:37:43.195 --> 00:37:45.270 left and then come back to it and then gone down. 00:37:45.270 --> 00:37:46.686 So overall we're doing an in order 00:37:46.686 --> 00:37:51.140 traversal of the compress trie, so it's linear time total. 00:37:56.271 --> 00:37:56.770 OK. 00:38:00.510 --> 00:38:02.084 Still not done though. 00:38:02.084 --> 00:38:03.750 What we have so far-- what have we done? 00:38:03.750 --> 00:38:06.450 We've taken our numbers, we've split them into digits. 00:38:06.450 --> 00:38:09.870 We randomly permuted the digits via hash function. 00:38:09.870 --> 00:38:12.660 That magically let us sort them in linear time. 00:38:12.660 --> 00:38:15.900 Now we've built a compress trie of that representation. 00:38:15.900 --> 00:38:19.290 The last thing to do is to fix the order of the children 00:38:19.290 --> 00:38:20.760 from each node. 00:38:20.760 --> 00:38:23.130 The tree is correct except for the order 00:38:23.130 --> 00:38:25.170 of the children in each node. 00:38:25.170 --> 00:38:30.060 So we've kind of sorted a little bit, but haven't fully sorted. 00:38:30.060 --> 00:38:34.760 So that's the last step. 00:38:34.760 --> 00:38:39.320 And we're going to do that with recursion. 00:38:43.430 --> 00:38:45.725 We're only going to recurse a constant number of times. 00:38:48.410 --> 00:38:49.550 Recursively sort. 00:39:20.970 --> 00:39:24.890 OK, I'm going to walk through this compress 00:39:24.890 --> 00:39:29.450 try trie we've built. And I guess it looks more like this. 00:39:29.450 --> 00:39:33.877 And for each edge I'm going to write down this key. 00:39:33.877 --> 00:39:35.460 First thing you're going to write down 00:39:35.460 --> 00:39:36.720 is the ID of the node. 00:39:36.720 --> 00:39:42.260 Let's say the ID is the in order traversal index of the node. 00:39:42.260 --> 00:39:43.380 OK, so boom. 00:39:43.380 --> 00:39:47.910 As we traverse the tree we can just increment this counter. 00:39:47.910 --> 00:39:51.390 Then we also want-- 00:39:51.390 --> 00:39:57.960 what I'm going to put in here is the actual chunk value 00:39:57.960 --> 00:39:59.940 that's at the top of the node. 00:39:59.940 --> 00:40:02.700 So something like this, or I guess this one. 00:40:02.700 --> 00:40:05.400 This one consists of two chunk values. 00:40:05.400 --> 00:40:09.352 So it's called c1, for chunk values, c1 and c2. 00:40:09.352 --> 00:40:11.310 It's the first chunk value and the second chunk 00:40:11.310 --> 00:40:13.451 value, first digit and the second digit. 00:40:13.451 --> 00:40:14.700 Just write down the first one. 00:40:18.062 --> 00:40:19.770 That's really the only one we care about. 00:40:19.770 --> 00:40:21.936 The second one wasn't branching so it's no big deal. 00:40:24.440 --> 00:40:26.240 OK, so there's some chunk value. 00:40:26.240 --> 00:40:30.380 Now, I don't want to write down the signature chunk value. 00:40:30.380 --> 00:40:32.180 I don't want to write down the hash value. 00:40:32.180 --> 00:40:37.040 I want the actual chunk, original chunk not the hash. 00:40:37.040 --> 00:40:39.590 That's why I write actual over here 00:40:39.590 --> 00:40:41.480 and then also, I'm going to write down 00:40:41.480 --> 00:40:42.780 the index of the edge. 00:40:42.780 --> 00:40:46.240 So this is the second edge, for example. 00:40:46.240 --> 00:40:49.560 OK, now this is in some sense the wrong index. 00:40:49.560 --> 00:40:52.190 And what I want to compute is where it's supposed to be. 00:40:52.190 --> 00:40:54.050 The edges have been permuted. 00:40:54.050 --> 00:40:56.090 So if I sort this thing, I claim that I'll 00:40:56.090 --> 00:41:00.390 find that all the desired permutations. 00:41:00.390 --> 00:41:02.060 OK, why? 00:41:02.060 --> 00:41:07.280 Because if I sort by this triple, first thing 00:41:07.280 --> 00:41:10.760 I sort by is by node ID, then I'm sorting by actual chunk 00:41:10.760 --> 00:41:12.119 value, then edge index. 00:41:12.119 --> 00:41:13.910 I don't care that I'm sorting by it, that's 00:41:13.910 --> 00:41:15.870 just coming along for the ride. 00:41:15.870 --> 00:41:20.270 So what this tells me is for each node 00:41:20.270 --> 00:41:23.150 it gives me the sorted value of the actual chunks, not 00:41:23.150 --> 00:41:24.560 the hash chunks. 00:41:24.560 --> 00:41:28.160 And then it tells me the order they used to be in. 00:41:28.160 --> 00:41:33.200 So this is a way to figure out the sorted permutation 00:41:33.200 --> 00:41:34.250 of these things. 00:41:34.250 --> 00:41:35.954 This gives you the old index. 00:41:35.954 --> 00:41:37.370 And so once you've sorted them you 00:41:37.370 --> 00:41:41.510 know that new correct order is in order by chunk. 00:41:41.510 --> 00:41:43.040 And so this gives you the inverse 00:41:43.040 --> 00:41:47.420 of the permutation over here, of where each edge should be. 00:41:47.420 --> 00:41:52.670 So once I've done this the last step is 00:41:52.670 --> 00:42:04.340 for each node permute the edges as given 00:42:04.340 --> 00:42:07.190 by this inverse permutation. 00:42:07.190 --> 00:42:08.118 Questions? 00:42:08.118 --> 00:42:09.659 AUDIENCE: Can you explain again that? 00:42:09.659 --> 00:42:11.330 You were basically sorting three times. 00:42:11.330 --> 00:42:14.756 ERIK DEMAINE: No, I'm sorting these single key values. 00:42:14.756 --> 00:42:16.880 The key value, the most significant part of the key 00:42:16.880 --> 00:42:17.780 is this. 00:42:17.780 --> 00:42:19.340 The next most significant part of the key is this. 00:42:19.340 --> 00:42:21.360 And the least significant part of the key is this. 00:42:21.360 --> 00:42:23.276 So imagine concatenating these things together 00:42:23.276 --> 00:42:24.400 into a single key value. 00:42:24.400 --> 00:42:26.540 Yeah, good question. 00:42:26.540 --> 00:42:29.270 What I'll talk about in a moment, how big this key is. 00:42:29.270 --> 00:42:32.390 But if I sort by the concatenation of those keys 00:42:32.390 --> 00:42:34.460 then what happens is I'm sorting by this 00:42:34.460 --> 00:42:35.632 and then sorting by this. 00:42:35.632 --> 00:42:37.340 But really, I'm doing one sort operation. 00:42:37.340 --> 00:42:39.620 This is one recursion. 00:42:39.620 --> 00:42:42.530 Other questions? 00:42:42.530 --> 00:42:44.200 OK. 00:42:44.200 --> 00:42:46.520 Do you see why this gives us what we need? 00:42:46.520 --> 00:42:49.500 For every node we want to sort all those chunks. 00:42:49.500 --> 00:42:51.806 And so this information is just so we 00:42:51.806 --> 00:42:53.180 can keep track of the permutation 00:42:53.180 --> 00:42:54.920 when we're just sorting by value and not 00:42:54.920 --> 00:42:56.500 keeping the permutation. 00:42:56.500 --> 00:42:58.650 This let's us keep track of it. 00:42:58.650 --> 00:43:01.760 But our real goal is to sort all the chunk values. 00:43:01.760 --> 00:43:04.430 And we add in the node ID so that we just 00:43:04.430 --> 00:43:06.740 learn for every node what we're supposed to do instead 00:43:06.740 --> 00:43:08.712 of globally what we need to do. 00:43:08.712 --> 00:43:10.670 Maybe you don't have to do this but it's easier 00:43:10.670 --> 00:43:13.010 to keep the node ID in there. 00:43:13.010 --> 00:43:17.004 OK, how many bits are in this key? 00:43:17.004 --> 00:43:18.920 So our whole point was to make chunks small so 00:43:18.920 --> 00:43:20.169 that the keys were small. 00:43:20.169 --> 00:43:22.460 But I claim this will be a little bit smaller than what 00:43:22.460 --> 00:43:23.730 we started with. 00:43:23.730 --> 00:43:26.300 So the node ID, well, there's only n nodes, 00:43:26.300 --> 00:43:29.510 so this is log n bits. 00:43:29.510 --> 00:43:35.090 Chunk value is, I guess, w divided by log 00:43:35.090 --> 00:43:38.630 to the epsilon n bits. 00:43:38.630 --> 00:43:42.070 Because there are a log to the epsilon n chunks, 00:43:42.070 --> 00:43:44.240 so it's w divided by log to the epsilon bits. 00:43:44.240 --> 00:43:50.390 The edge index, how many different things were there? 00:43:54.822 --> 00:43:57.090 Is it also w divided by log to the epsilon? 00:43:57.090 --> 00:43:59.524 I guess so. 00:43:59.524 --> 00:44:00.440 Depends how you count. 00:44:04.390 --> 00:44:06.800 Let me think for a second how these edge indices should 00:44:06.800 --> 00:44:07.300 be stored. 00:44:12.140 --> 00:44:14.540 I think I'd like to store the edge indices-- 00:44:14.540 --> 00:44:20.150 I don't want to store the absent down pointers. 00:44:20.150 --> 00:44:23.680 The way that I built this tree, this compress trie, 00:44:23.680 --> 00:44:25.920 I was always adding things to the right. 00:44:25.920 --> 00:44:28.554 So it's like OK, I'm storing here. 00:44:28.554 --> 00:44:30.470 Here's the first child, here's the next child, 00:44:30.470 --> 00:44:32.344 here's the next child, here's the next child. 00:44:32.344 --> 00:44:33.450 I just keep appending on. 00:44:33.450 --> 00:44:35.570 It's like an array that you can grow to the right, which 00:44:35.570 --> 00:44:37.460 you should know how to do in constant time, 00:44:37.460 --> 00:44:39.168 even though linked list is probably fine. 00:44:41.342 --> 00:44:42.800 If there are a bunch of values here 00:44:42.800 --> 00:44:44.258 that are just null pointers I don't 00:44:44.258 --> 00:44:46.470 want to store those items in the array. 00:44:46.470 --> 00:44:50.360 So in fact, the maximum number of children of a node is n. 00:44:50.360 --> 00:44:53.190 I mean, at the worst case you have branching everywhere. 00:44:53.190 --> 00:45:03.050 So this is only log n bits to store the index in that array. 00:45:03.050 --> 00:45:05.240 So it's an extra level of compression. 00:45:05.240 --> 00:45:13.150 I guess I didn't mention don't store null pointers, 00:45:13.150 --> 00:45:16.040 just skip them. 00:45:16.040 --> 00:45:19.510 And that's easy to do because you're always just appending. 00:45:19.510 --> 00:45:24.300 And in the end what do I need to do here? 00:45:24.300 --> 00:45:33.010 Step seven is in order traversal of the compress trie. 00:45:37.180 --> 00:45:43.654 And I output all the leaves I get in order. 00:45:43.654 --> 00:45:45.820 And the point is we computed the compress trie which 00:45:45.820 --> 00:45:48.910 has the rough correct topological structure. 00:45:48.910 --> 00:45:50.710 Then we fix the order within each node, 00:45:50.710 --> 00:45:53.320 now we have the correctly sorted compress trie. 00:45:53.320 --> 00:45:55.980 And so you're just doing in order traversal, output 00:45:55.980 --> 00:45:57.610 the leave you get in order. 00:45:57.610 --> 00:46:00.690 Boom, we've sorted all the items. 00:46:00.690 --> 00:46:01.315 A little crazy. 00:46:03.887 --> 00:46:06.220 So if all we need to do in the end is in order traversal 00:46:06.220 --> 00:46:08.553 then it's fine we don't need to store the null pointers. 00:46:08.553 --> 00:46:10.920 So we don't need to be able to search in this trie, 00:46:10.920 --> 00:46:11.710 so this is enough. 00:46:14.590 --> 00:46:16.090 Clear? 00:46:16.090 --> 00:46:17.950 OK, the remaining question is how 00:46:17.950 --> 00:46:19.970 expensive is this recursion? 00:46:19.970 --> 00:46:23.170 So I computed the number of bits basically w 00:46:23.170 --> 00:46:27.020 divided by log to the epsilon n, plus order log n. 00:46:27.020 --> 00:46:29.050 That order log n won't hurt us. 00:46:29.050 --> 00:46:32.660 Now, we started with integers of size w, 00:46:32.660 --> 00:46:35.740 the total size here was w. 00:46:35.740 --> 00:46:39.850 Now we have integers that their size is w divided 00:46:39.850 --> 00:46:40.910 by log to the epsilon n. 00:46:40.910 --> 00:46:41.868 So we've made progress. 00:46:45.010 --> 00:46:46.520 Cool. 00:46:46.520 --> 00:46:47.645 How much progress? 00:46:53.380 --> 00:46:54.210 Over here maybe. 00:47:01.710 --> 00:47:05.280 After, let's say, 1 over epsilon plus 1 00:47:05.280 --> 00:47:17.230 recursions we will have reached b will 00:47:17.230 --> 00:47:27.110 have decreased to order log n. 00:47:27.110 --> 00:47:36.350 I guess, like 1 over epsilon log n plus w over log 00:47:36.350 --> 00:47:39.800 to the 1 plus epsilon n. 00:47:47.480 --> 00:47:48.940 OK. 00:47:48.940 --> 00:47:50.850 Right, if we take w and every time we 00:47:50.850 --> 00:47:54.630 divide by log to the epsilon n, then after 1 over epsilon times 00:47:54.630 --> 00:47:57.700 we have divided by log n, get rid of epsilon. 00:47:57.700 --> 00:47:59.910 Do it one more time, we get w divided 00:47:59.910 --> 00:48:03.480 by log to the 1 plus epsilon n. 00:48:03.480 --> 00:48:05.720 And at this point we're in good shape. 00:48:09.600 --> 00:48:11.700 If we get the size of our integers 00:48:11.700 --> 00:48:15.360 to be only w divided by log to the 1 plus epsilon n, 00:48:15.360 --> 00:48:18.450 then we can use packed sorting. 00:48:18.450 --> 00:48:20.310 Because packed sorting, conveniently it's 00:48:20.310 --> 00:48:22.410 the black box at the moment, says well, 00:48:22.410 --> 00:48:25.710 as long as w divided by b is a little bigger than log n. 00:48:25.710 --> 00:48:28.260 So here we've made it quite a bit bigger than log n. 00:48:28.260 --> 00:48:31.560 I mean, log to the 1 plus epsilon n bigger, 00:48:31.560 --> 00:48:33.270 then we can use packed sorting. 00:48:37.680 --> 00:48:44.550 Reach base case, and then use packed sorting. 00:48:50.740 --> 00:48:53.210 OK, the only thing is there's this log n. 00:48:53.210 --> 00:48:56.890 Now, epsilon is a constant here, so 1 over epsilon is constant. 00:48:56.890 --> 00:48:58.740 So this is just order log n. 00:48:58.740 --> 00:49:00.280 Order log n doesn't hurt you. 00:49:00.280 --> 00:49:03.030 I mean, if this happened to dominate this 00:49:03.030 --> 00:49:06.300 then you're saying your values are order log n bits long, 00:49:06.300 --> 00:49:08.740 and so you use radex sort. 00:49:08.740 --> 00:49:10.612 So that's not going to dominate. 00:49:10.612 --> 00:49:12.070 In fact, you can prove in this case 00:49:12.070 --> 00:49:14.528 it's not going to dominate. w is at least log to the 2 plus 00:49:14.528 --> 00:49:15.500 epsilon. 00:49:15.500 --> 00:49:17.190 So you take w divided by log to 1 00:49:17.190 --> 00:49:20.010 plus epsilon, that's still at least log n. 00:49:20.010 --> 00:49:21.600 So this will always dominate. 00:49:27.360 --> 00:49:31.500 OK, so we did all this work and all 00:49:31.500 --> 00:49:35.142 we did was reduce our word size by a log to the epsilon factor. 00:49:35.142 --> 00:49:36.600 But that's enough, because after we 00:49:36.600 --> 00:49:40.310 do that 1 over epsilon plus 1 times 00:49:40.310 --> 00:49:42.990 the words are small enough, or the items we're sorting 00:49:42.990 --> 00:49:45.750 are small enough, that we can use packed sorting. 00:49:45.750 --> 00:49:47.700 And boom, we're done. 00:49:47.700 --> 00:49:52.140 OK, that is the beauty of signature sort. 00:49:52.140 --> 00:49:54.780 Any questions about that? 00:49:54.780 --> 00:49:56.070 Definitely not easy. 00:49:59.020 --> 00:50:01.640 But, it works. 00:50:01.640 --> 00:50:04.594 The crazy idea is, well, decompose into these digits. 00:50:04.594 --> 00:50:06.760 And when we decompose into log to the epsilon digits 00:50:06.760 --> 00:50:08.270 because that would be enough. 00:50:08.270 --> 00:50:10.270 If we can get things down to a single digit life 00:50:10.270 --> 00:50:12.460 would be easy. 00:50:12.460 --> 00:50:16.550 Which I guess is kind of like Van Emde Boas, right. 00:50:16.550 --> 00:50:18.790 It's like our lower bounds from last time. 00:50:18.790 --> 00:50:19.900 I don't know. 00:50:19.900 --> 00:50:22.960 If we can make it to only one digit mattered then 00:50:22.960 --> 00:50:25.750 we could afford to do it because each digit is quite a bit 00:50:25.750 --> 00:50:26.290 smaller. 00:50:26.290 --> 00:50:30.790 It's smaller by a factor of log to the epsilon n. 00:50:30.790 --> 00:50:32.700 So we did that splitting and then there 00:50:32.700 --> 00:50:36.200 was this observation that most digit values aren't used. 00:50:36.200 --> 00:50:39.520 So if we hash then things got a lot smaller. 00:50:39.520 --> 00:50:41.260 Then we can afford to pack sort. 00:50:41.260 --> 00:50:43.455 Except that hash didn't preserve the order. 00:50:43.455 --> 00:50:44.830 How didn't it preserve the order? 00:50:44.830 --> 00:50:46.660 Well, when you look at the compress trie 00:50:46.660 --> 00:50:48.430 you see how it got messed up. 00:50:48.430 --> 00:50:50.210 Each node got messed up individually, 00:50:50.210 --> 00:50:52.535 but overall the structure of the trie was correct. 00:50:52.535 --> 00:50:54.910 So we did all this work to get the structure of the trie. 00:50:54.910 --> 00:50:56.950 Then we had to fix the node order. 00:50:56.950 --> 00:50:59.080 But that's really just sorting a single digit. 00:50:59.080 --> 00:51:01.900 And yeah, we have to do it for all nodes but you add them up. 00:51:01.900 --> 00:51:05.200 It's still only n items that we need to sort. 00:51:05.200 --> 00:51:08.500 So I wrote that b went down. 00:51:08.500 --> 00:51:10.515 n stays the same. 00:51:10.515 --> 00:51:12.140 The number of things we have to sort of 00:51:12.140 --> 00:51:15.375 stays the same, which is fine. 00:51:15.375 --> 00:51:17.190 We're not trying to reduce n, we're 00:51:17.190 --> 00:51:19.030 just trying to reduce the size of our items 00:51:19.030 --> 00:51:20.700 to be a little bit smaller than w 00:51:20.700 --> 00:51:22.770 so then we can use packed sorting. 00:51:22.770 --> 00:51:26.800 OK, I think I've gone over this algorithm enough times. 00:51:26.800 --> 00:51:29.320 The next step is packed sorting. 00:51:32.290 --> 00:51:36.940 If we have integers that are much smaller than our word, 00:51:36.940 --> 00:51:39.360 how do we sort in linear time? 00:51:39.360 --> 00:51:43.140 And this is going to be essentially a fancy merge sort. 00:52:09.284 --> 00:52:10.575 This is done in the same paper. 00:52:26.560 --> 00:52:30.910 Before we said omega, I'm going to essentially make omega 00:52:30.910 --> 00:52:32.057 a little bit bigger than 2. 00:52:32.057 --> 00:52:33.640 So I want to assume that w is at least 00:52:33.640 --> 00:52:36.910 twice b plus 1 log n log log n. 00:52:36.910 --> 00:52:38.860 That's just to make this convenient, 00:52:38.860 --> 00:52:42.042 but of course it doesn't really matter what the constants are. 00:52:42.042 --> 00:52:43.750 This will make everything fit in one word 00:52:43.750 --> 00:52:46.410 though, instead of a constant number of words. 00:52:46.410 --> 00:53:03.000 OK, step zero of this algorithm is the packing part. 00:53:07.260 --> 00:53:09.780 We can fit more than one element into each word. 00:53:09.780 --> 00:53:12.060 So put them in. 00:53:12.060 --> 00:53:13.860 Now, we have linear time overall. 00:53:13.860 --> 00:53:16.550 This is easy to do. 00:53:16.550 --> 00:53:17.940 You have some word already. 00:53:20.580 --> 00:53:23.940 Take those items, shift them left or in the next item, 00:53:23.940 --> 00:53:25.530 shift them left or in the next item, 00:53:25.530 --> 00:53:27.440 shift them left or in the next item. 00:53:27.440 --> 00:53:30.460 OK, linear time. 00:53:30.460 --> 00:53:33.610 So in the end what it's going to look like-- 00:53:33.610 --> 00:53:36.810 oh, I also want to leave a 0 bit in between each of them. 00:53:36.810 --> 00:53:39.390 So we're going to have b. 00:53:39.390 --> 00:53:53.150 This is going to be x1 here, x2, xn. 00:53:53.150 --> 00:53:55.171 Each of these is b bits long. 00:53:55.171 --> 00:53:56.420 And then we have one more bit. 00:53:58.940 --> 00:54:00.710 And in fact, the way we set things up this 00:54:00.710 --> 00:54:02.074 is only half the word. 00:54:02.074 --> 00:54:03.740 So there's another half over here, which 00:54:03.740 --> 00:54:06.470 is going to be all 0 for now. 00:54:06.470 --> 00:54:08.890 So we have one word contains-- 00:54:08.890 --> 00:54:10.680 sorry, this is not n items. 00:54:10.680 --> 00:54:14.929 This is going to be log n log log n. 00:54:14.929 --> 00:54:16.220 Can't fit them all in one word. 00:54:16.220 --> 00:54:18.194 That would make life really easy. 00:54:18.194 --> 00:54:20.360 You can fit log n log log n items in the first word. 00:54:20.360 --> 00:54:22.940 Then the next word is the next log n 00:54:22.940 --> 00:54:25.260 log log n items and so on. 00:54:25.260 --> 00:54:26.480 Linear time to do that. 00:54:30.170 --> 00:54:33.472 Next step, this is an algorithm. 00:54:33.472 --> 00:54:34.430 It's not really a step. 00:54:34.430 --> 00:54:38.320 I should say these are things that we're able to do, 00:54:38.320 --> 00:54:41.144 and we're going to combine them in different ways. 00:54:41.144 --> 00:54:42.560 So the next thing to observe, this 00:54:42.560 --> 00:54:46.070 is sort of a bottom up perspective. 00:54:46.070 --> 00:54:48.320 So first thing we can do is pack things into word. 00:54:48.320 --> 00:54:52.000 Next thing we can do is merge two sorted words. 00:54:52.000 --> 00:54:55.670 Let's say they each have k items and k is at most log n log log 00:54:55.670 --> 00:54:56.170 n. 00:55:04.480 --> 00:55:15.575 And I want to merge them into one sorted word with two k 00:55:15.575 --> 00:55:16.075 elements. 00:55:18.970 --> 00:55:22.060 So I have two of these things and I'll merge them 00:55:22.060 --> 00:55:25.244 in sorted order. 00:55:25.244 --> 00:55:27.160 And when I did this packing, nothing's sorted. 00:55:27.160 --> 00:55:29.170 But don't worry about that at the moment. 00:55:29.170 --> 00:55:31.690 Just if I had two these words and suppose 00:55:31.690 --> 00:55:34.990 they were already sorted, then I want 00:55:34.990 --> 00:55:39.610 to merge them and make them still sorted. 00:55:39.610 --> 00:55:41.860 OK, I want x1 to be less than x2 and so on. 00:55:44.470 --> 00:55:45.895 How much time? 00:55:45.895 --> 00:55:48.340 It'd be cool if you could do this in constant time 00:55:48.340 --> 00:55:49.591 on a transdichotomous rem. 00:55:49.591 --> 00:55:51.340 You could do anything on a constant number 00:55:51.340 --> 00:55:52.532 of words in constant time. 00:55:52.532 --> 00:55:54.240 We're not going to achieve constant time, 00:55:54.240 --> 00:55:55.880 but we don't need to. 00:55:55.880 --> 00:55:57.960 We're going to do order log k time. 00:56:01.235 --> 00:56:03.610 ERIK DEMAINE: This is the hardest step of packed sorting, 00:56:03.610 --> 00:56:06.790 and I'm not going to tell you how to do it until later. 00:56:06.790 --> 00:56:08.890 I will tell you, don't worry. 00:56:08.890 --> 00:56:13.210 But this is going to require bitonic sorting. 00:56:13.210 --> 00:56:15.650 So I have to delay it a little bit. 00:56:15.650 --> 00:56:17.320 Suppose you had that for now. 00:56:17.320 --> 00:56:19.030 Let me tell you how the rest is easy. 00:56:25.900 --> 00:56:28.860 So many black boxes, but they're all filled in in this lecture. 00:56:31.660 --> 00:56:37.060 OK, next thing you might ask is, well, 00:56:37.060 --> 00:56:38.860 how do I get a sorted word? 00:56:38.860 --> 00:56:42.190 I see how to get a single word that has log n log log n items. 00:56:42.190 --> 00:56:44.640 But how would I sort those items? 00:56:44.640 --> 00:56:46.360 Use merge sort. 00:56:46.360 --> 00:56:47.410 Here's a merger. 00:56:47.410 --> 00:56:49.599 You have a way to merge two sorted lists, 00:56:49.599 --> 00:56:51.140 so use merge sort, that will give you 00:56:51.140 --> 00:56:53.270 a way to sort the whole thing. 00:56:53.270 --> 00:56:58.060 So I'm going to merge sort k equals log n log log n 00:56:58.060 --> 00:57:06.300 items into one sorted word. 00:57:13.050 --> 00:57:15.690 So I start out with a word that's unsorted. 00:57:15.690 --> 00:57:17.490 I do merge sort. 00:57:17.490 --> 00:57:19.884 And I sort them. 00:57:19.884 --> 00:57:21.550 So this is not the whole sorting problem 00:57:21.550 --> 00:57:25.300 but it's the one word sorting problem. 00:57:25.300 --> 00:57:27.060 So how long does this take? 00:57:27.060 --> 00:57:30.020 Well, it's usual merge sort recurrence, 00:57:30.020 --> 00:57:34.095 but then it's plus how long it takes me to split the word. 00:57:34.095 --> 00:57:35.470 Well, splitting the word is easy, 00:57:35.470 --> 00:57:38.310 you just mask out the low end or the high end 00:57:38.310 --> 00:57:40.140 and then maybe shift over. 00:57:40.140 --> 00:57:42.900 So in constant time I can get the left half 00:57:42.900 --> 00:57:46.230 and the right half of the array, which is the word. 00:57:46.230 --> 00:57:47.850 Then the hard part is merging them. 00:57:47.850 --> 00:57:54.468 And merging them is 1, and that takes log k. 00:57:54.468 --> 00:57:57.710 So this is step one. 00:57:57.710 --> 00:57:59.010 That's our merger. 00:57:59.010 --> 00:58:00.560 And so what does this solve to? 00:58:00.560 --> 00:58:03.600 Well, probably easiest to draw the recursion tree. 00:58:03.600 --> 00:58:05.550 So at the root we pay log k. 00:58:05.550 --> 00:58:13.890 Then we have 2 times log k over 2, log k over 2. 00:58:16.770 --> 00:58:20.160 So at this level we're paying log k. 00:58:20.160 --> 00:58:26.280 At this level we're paying 2 log k minus 1, I guess. 00:58:28.880 --> 00:58:30.720 I'm just going to call this maybe-- 00:58:30.720 --> 00:58:33.480 yeah, I do need to do that. 00:58:33.480 --> 00:58:36.030 I mean everything is constant size so you pay a constant. 00:58:36.030 --> 00:58:39.200 How many leaves are there at the bottom level? 00:58:39.200 --> 00:58:42.260 k of them. 00:58:42.260 --> 00:58:44.090 OK, so at the root we're paying log k, 00:58:44.090 --> 00:58:45.750 at the leaves we're paying k. 00:58:45.750 --> 00:58:47.310 This is roughly geometric. 00:58:47.310 --> 00:58:50.540 There's this minus 1, but it's geometric enough 00:58:50.540 --> 00:58:54.210 and when you add up all these levels it's dominated by k. 00:58:54.210 --> 00:58:56.872 So this ends up being order k time. 00:58:59.450 --> 00:59:03.860 The minus 1 here turns out not to matter much. 00:59:03.860 --> 00:59:04.710 Cool. 00:59:04.710 --> 00:59:08.110 So this linear time sorting on log n log log n items. 00:59:08.110 --> 00:59:10.050 Everything it fits in a word. 00:59:10.050 --> 00:59:13.060 Cool, but we're not done yet. 00:59:13.060 --> 00:59:16.320 We need another level. 00:59:16.320 --> 00:59:19.455 The rest is pretty easy. 00:59:19.455 --> 00:59:21.010 It's just scaling up. 00:59:54.660 --> 00:59:56.540 OK, let's see. 00:59:56.540 --> 01:00:00.530 At this point we're going to assume that all of our words 01:00:00.530 --> 01:00:02.150 are completely full. 01:00:02.150 --> 01:00:05.270 So this merging and stuff was in order to fill up words 01:00:05.270 --> 01:00:07.310 and to make those words sorted. 01:00:07.310 --> 01:00:10.640 But once we've done that for each cluster of log n 01:00:10.640 --> 01:00:13.130 log log n items, now we can assume each word 01:00:13.130 --> 01:00:14.650 is sorted and completely full. 01:00:14.650 --> 01:00:18.150 It has exactly k elements in it. 01:00:18.150 --> 01:00:25.540 So each of these r sorted words has k elements. 01:00:25.540 --> 01:00:28.100 OK, so now the issue is things don't fit in a word. 01:00:28.100 --> 01:00:30.410 And so now suppose they fit in r sorted words. 01:00:30.410 --> 01:00:35.120 So supposed we've already made a sorted list of r times k items 01:00:35.120 --> 01:00:38.720 which fit in r words, each with k items in it. 01:00:38.720 --> 01:00:40.760 Then I want to take two of these lists 01:00:40.760 --> 01:00:42.590 and merge them together and get one 01:00:42.590 --> 01:00:44.620 sorted list of sorted words. 01:00:44.620 --> 01:00:50.570 So it's 2 rk items that are distributed into 2r words, 01:00:50.570 --> 01:00:53.960 each of size k. 01:00:53.960 --> 01:00:57.750 How do I do that merge in linear time? 01:00:57.750 --> 01:01:03.700 And I only need to do it in r log k time, only. 01:01:03.700 --> 01:01:04.550 Yeah, question? 01:01:04.550 --> 01:01:08.860 AUDIENCE: I'm assuming k is the number of items per word? 01:01:08.860 --> 01:01:12.230 ERIK DEMAINE: Yeah, k is this from now on. 01:01:12.230 --> 01:01:16.129 And there are exactly k elements per word. 01:01:16.129 --> 01:01:18.170 We've already filled up the word so at this point 01:01:18.170 --> 01:01:19.320 we can assume that. 01:01:19.320 --> 01:01:21.427 AUDIENCE: But it's exactly equal, not less than. 01:01:21.427 --> 01:01:22.260 ERIK DEMAINE: Right. 01:01:22.260 --> 01:01:23.132 AUDIENCE: Because in one-- 01:01:23.132 --> 01:01:24.830 ERIK DEMAINE: Yeah, over here it's less than or equal to. 01:01:24.830 --> 01:01:26.121 I know it's a little confusing. 01:01:26.121 --> 01:01:29.280 I use k for different-- call this k prime. 01:01:29.280 --> 01:01:33.080 OK, and at this point we've filled up to this value of k. 01:01:33.080 --> 01:01:35.570 Now everything has exactly log n log log n items. 01:01:35.570 --> 01:01:38.120 Thanks. 01:01:38.120 --> 01:01:40.070 OK, let's see. 01:01:42.650 --> 01:01:45.590 A regular merging algorithm would take r times k time 01:01:45.590 --> 01:01:47.970 because they're r times k items. 01:01:47.970 --> 01:01:51.980 But we have this merger over here, which will 01:01:51.980 --> 01:01:55.040 merge two words in log k time. 01:01:55.040 --> 01:01:58.250 So that's why it's going to be r log k. 01:01:58.250 --> 01:01:59.300 What's the picture? 01:01:59.300 --> 01:02:00.450 Well, let me draw it. 01:02:16.900 --> 01:02:21.080 We have a list of words. 01:02:21.080 --> 01:02:24.040 A list of words. 01:02:24.040 --> 01:02:24.880 How do we merge? 01:02:24.880 --> 01:02:27.880 Well, we take the first two words, merge them. 01:02:27.880 --> 01:02:30.620 When we merge them and we know how to do this n log k time, 01:02:30.620 --> 01:02:33.220 that's step 1. 01:02:33.220 --> 01:02:34.954 When we merge them we get two words, 01:02:34.954 --> 01:02:35.995 they just are now sorted. 01:02:39.100 --> 01:02:41.470 So we've got these two guys sorted 01:02:41.470 --> 01:02:43.510 and then there's the rest. 01:02:43.510 --> 01:02:45.610 I'd like to say output these and repeat, 01:02:45.610 --> 01:02:48.640 but that wouldn't be quite correct. 01:02:48.640 --> 01:02:54.979 Well, these guys I can output because I've got k items here, 01:02:54.979 --> 01:02:55.520 k items here. 01:02:55.520 --> 01:02:57.590 These are the k smallest among all of them. 01:02:57.590 --> 01:03:00.340 Those must be the overall smallest k items. 01:03:00.340 --> 01:03:02.050 Because I compared k with k. 01:03:02.050 --> 01:03:04.930 These higher k items, I don't know. 01:03:04.930 --> 01:03:06.560 They might be good or bad. 01:03:06.560 --> 01:03:11.971 So I have to put this high part back into one of these lists. 01:03:11.971 --> 01:03:13.720 You have to think a little bit about which 01:03:13.720 --> 01:03:14.590 list to put it into. 01:03:14.590 --> 01:03:16.690 Turns out if you look at the max item, 01:03:16.690 --> 01:03:19.357 say where did this come from, list one or list two? 01:03:19.357 --> 01:03:20.440 You put it into that list. 01:03:20.440 --> 01:03:22.815 Let's say this came from list two. 01:03:22.815 --> 01:03:25.970 And you put it back into list two and then repeat. 01:03:25.970 --> 01:03:32.710 OK, so in log k time here we have output k items. 01:03:32.710 --> 01:03:36.790 And so we end up with an r log k running time overall. 01:03:36.790 --> 01:03:39.490 OK, so it's like regular merge algorithm, 01:03:39.490 --> 01:03:43.420 except we use step one to merge words, 01:03:43.420 --> 01:03:45.690 and then there's a little bit of work. 01:03:45.690 --> 01:03:47.470 This is like old merging algorithm, right? 01:03:47.470 --> 01:03:50.770 You take the min and you put the max back in. 01:03:50.770 --> 01:03:55.040 Except now the max is a whole word. 01:03:55.040 --> 01:03:58.030 OK that's how we merge. 01:03:58.030 --> 01:03:59.260 Now, why did we do a merger? 01:03:59.260 --> 01:04:01.770 Because next thing we do is merge sort. 01:04:20.250 --> 01:04:22.160 And this is the overall algorithm. 01:04:22.160 --> 01:04:23.070 It's one merge sort. 01:04:23.070 --> 01:04:24.750 We're going to use this is the merger 01:04:24.750 --> 01:04:26.650 and we're going to use this as the base case. 01:04:26.650 --> 01:04:30.110 So it's actually two recursive levels of merge sort. 01:04:30.110 --> 01:04:33.390 Merge sort is recursive within step four for a while, 01:04:33.390 --> 01:04:35.751 until we get down to the level of a single word. 01:04:35.751 --> 01:04:38.250 Then we use this thing to deal with things in a single word. 01:04:42.580 --> 01:04:49.230 So we end up with t of n is 2 times t of n over 2 01:04:49.230 --> 01:04:55.890 plus n over k log k. 01:04:55.890 --> 01:05:00.000 And base case of t of k is order k. 01:05:07.350 --> 01:05:08.670 Why is it all this? 01:05:08.670 --> 01:05:11.750 This thing is r. 01:05:15.450 --> 01:05:20.430 So we have r log k to do a merge. r here is n over k. 01:05:20.430 --> 01:05:21.470 Why is it n over k? 01:05:24.250 --> 01:05:26.190 Because n is the total number of items. 01:05:26.190 --> 01:05:29.190 We had r times k items, so n equals rk. 01:05:29.190 --> 01:05:31.040 So r equals n over k. 01:05:31.040 --> 01:05:32.177 That's that. 01:05:32.177 --> 01:05:33.260 So this is our merge cost. 01:05:33.260 --> 01:05:38.160 This is the cost of 3, this is the cost of 2. 01:05:38.160 --> 01:05:41.920 If we have only log n log log n items, 01:05:41.920 --> 01:05:48.150 so k items, then we can sort in linear time. 01:05:48.150 --> 01:05:50.332 So we just need to solve this recurrence. 01:05:52.990 --> 01:05:56.820 Well, how's it going to work? 01:05:56.820 --> 01:06:00.770 It's going to be kind of like merge sort, but not quite. 01:06:00.770 --> 01:06:01.270 Question? 01:06:03.920 --> 01:06:06.170 AUDIENCE: When you say-- so you have-- it's definitely 01:06:06.170 --> 01:06:08.620 merging two lists of sorted words 01:06:08.620 --> 01:06:11.070 into one sorted list of two words. 01:06:11.070 --> 01:06:12.050 [INAUDIBLE]? 01:06:16.950 --> 01:06:18.910 You may have to do that, right? 01:06:21.232 --> 01:06:22.440 ERIK DEMAINE: Say that again. 01:06:22.440 --> 01:06:24.316 AUDIENCE: So you have two sorted-- 01:06:24.316 --> 01:06:27.425 two sorted lists of R sorted words. 01:06:27.425 --> 01:06:28.800 ERIK DEMAINE: The words are going 01:06:28.800 --> 01:06:31.530 to change, which elements are in which words are changing. 01:06:31.530 --> 01:06:36.300 But that's thanks to step one, for example. 01:06:36.300 --> 01:06:38.640 I mean, step one can merge any two sorted words 01:06:38.640 --> 01:06:41.070 and make one sorted word. 01:06:41.070 --> 01:06:45.180 Or actually, one sorted word. 01:06:45.180 --> 01:06:48.360 In our case, we're going to get two sorted words. 01:06:48.360 --> 01:06:49.840 That was the picture over here. 01:06:49.840 --> 01:06:51.510 I merge the two guys. 01:06:51.510 --> 01:06:54.720 I could represent it as one word that's kind of double full, 01:06:54.720 --> 01:06:57.420 and then I can split that into two words. 01:06:57.420 --> 01:06:59.190 So we again need the split operation, 01:06:59.190 --> 01:07:04.620 but split is just a mask these guys out and you 01:07:04.620 --> 01:07:05.760 get the high part. 01:07:05.760 --> 01:07:07.090 This is all 0. 01:07:07.090 --> 01:07:09.480 Then you shift it over to the left, 01:07:09.480 --> 01:07:14.130 now you've got the high word over the single word. 01:07:14.130 --> 01:07:17.230 So they're shuffling within those two words, 01:07:17.230 --> 01:07:20.980 or within one word, essentially, via this procedure, 01:07:20.980 --> 01:07:22.859 which we haven't covered. 01:07:22.859 --> 01:07:24.400 So everything is going to mix around. 01:07:24.400 --> 01:07:27.064 But then there's the low items and the high items. 01:07:27.064 --> 01:07:28.980 And so all the items in here are less than all 01:07:28.980 --> 01:07:32.080 of the items in here. 01:07:32.080 --> 01:07:33.630 Other questions? 01:07:33.630 --> 01:07:34.980 Cool. 01:07:34.980 --> 01:07:37.150 So I need to solve this recurrence. 01:07:37.150 --> 01:07:40.040 So let's draw another recursion tree. 01:07:40.040 --> 01:07:42.250 That's the east way to solve recurrences. 01:07:42.250 --> 01:07:44.970 Root is n over k log k. 01:07:44.970 --> 01:07:48.750 Next level down, see n got divided by 2. 01:07:48.750 --> 01:07:57.690 So it's going to be 1/2 n over k log k, 1/2 n over k log k, 01:07:57.690 --> 01:07:59.460 because the n gets divided by 2. 01:07:59.460 --> 01:08:01.410 So if I add up everything on this level, 01:08:01.410 --> 01:08:04.080 I get n over k log k. 01:08:04.080 --> 01:08:06.870 And in general as I go down, all levels 01:08:06.870 --> 01:08:10.080 are going to sum to this same value, n over k log k. 01:08:10.080 --> 01:08:13.335 So up here we are going to have a cost of n 01:08:13.335 --> 01:08:18.120 over k log k times the number of levels. 01:08:18.120 --> 01:08:20.069 Now, this is where it's a little tricky. 01:08:20.069 --> 01:08:22.080 It's not log n levels like regular merge sort. 01:08:22.080 --> 01:08:25.010 We stop at level k. 01:08:25.010 --> 01:08:28.050 So it's going to be log of n over k. 01:08:33.050 --> 01:08:35.370 That's the number of levels we get to. 01:08:35.370 --> 01:08:40.939 Now at the leaf level finally we have things of size order k, 01:08:40.939 --> 01:08:44.020 and we only pay order k. 01:08:44.020 --> 01:08:46.470 Great. 01:08:46.470 --> 01:08:48.915 And then how many leaves are there? 01:08:48.915 --> 01:08:50.460 There's n over k leaves. 01:08:56.230 --> 01:08:56.729 OK. 01:08:56.729 --> 01:09:00.050 So, that's an order n cost. 01:09:00.050 --> 01:09:05.640 This is order k, that's n over k, order n. 01:09:05.640 --> 01:09:08.715 What's this thing? 01:09:08.715 --> 01:09:09.779 Well, what's k? 01:09:09.779 --> 01:09:12.840 k was log n log log n. 01:09:12.840 --> 01:09:16.680 So this is n divided by log n log log n. 01:09:16.680 --> 01:09:20.130 This, therefore, is approximately log log n. 01:09:20.130 --> 01:09:22.330 So that log log n cancels this one. 01:09:22.330 --> 01:09:26.065 This is approximately log n, so that cancels with this one. 01:09:26.065 --> 01:09:29.340 So this is order n. 01:09:29.340 --> 01:09:32.104 That's why we chose these values. 01:09:32.104 --> 01:09:34.229 So this n over k was not actually that significant. 01:09:34.229 --> 01:09:35.700 We could have called that n. 01:09:35.700 --> 01:09:40.140 What is important is that this recursion doesn't work when 01:09:40.140 --> 01:09:42.240 you get down to a single word. 01:09:42.240 --> 01:09:43.800 Can't do the same strategy. 01:09:43.800 --> 01:09:47.700 You need to use a different merger for single words 01:09:47.700 --> 01:09:49.800 because here we have to do bit tricks. 01:09:49.800 --> 01:09:51.180 For this merge strategy we didn't 01:09:51.180 --> 01:09:53.305 have to do big tricks because everything was bigger 01:09:53.305 --> 01:09:55.856 than a word. 01:09:55.856 --> 01:09:57.480 The reason why we have these two levels 01:09:57.480 --> 01:10:01.050 is not to make the time fast, it's just 01:10:01.050 --> 01:10:02.640 to make the algorithm work. 01:10:02.640 --> 01:10:05.880 I mean, to define the algorithm there's within a word merging 01:10:05.880 --> 01:10:08.880 and over many words merging. 01:10:08.880 --> 01:10:11.910 The hard part is this one, within a word merging. 01:10:11.910 --> 01:10:14.650 So that's our next goal. 01:10:14.650 --> 01:10:19.560 But if we could do that we get linear time sorting for 01:10:19.560 --> 01:10:22.530 packed sorting when you can fit log log log n things in a word. 01:10:22.530 --> 01:10:24.280 Any questions about packed sorting? 01:10:27.180 --> 01:10:28.160 Good. 01:10:28.160 --> 01:10:33.800 So, last thing is how do we merge two sorted words 01:10:33.800 --> 01:10:34.790 into one sorted word. 01:10:37.562 --> 01:10:39.770 And this is where we're going to use bitonic sorting. 01:10:39.770 --> 01:10:42.394 See, it says right here, bitonic sort for merging sorted words. 01:10:42.394 --> 01:10:45.500 Here we know what to do. 01:10:45.500 --> 01:10:47.048 Much easier in hindsight. 01:10:51.660 --> 01:10:53.660 So I'm going to take a little diversion tell you 01:10:53.660 --> 01:10:56.405 about bitonic sorting, briefly. 01:11:01.240 --> 01:11:04.820 Because it's NCLRS I don't want to spend too much time on it, 01:11:04.820 --> 01:11:07.740 but it is something we need. 01:11:07.740 --> 01:11:09.380 So you need to know how it works. 01:11:14.310 --> 01:11:17.810 So first thing is the idea of a bitonic sequence. 01:11:17.810 --> 01:11:26.150 This is going to be a cyclic shift of a uni-modal sequence. 01:11:26.150 --> 01:11:32.900 So a non-decreasing plus a non-increasing sequence. 01:11:36.110 --> 01:11:39.650 So I want non-decreasing and then non-increasing. 01:11:39.650 --> 01:11:41.530 So this is a bitonic sequence. 01:11:41.530 --> 01:11:44.210 I think you know what this means. 01:11:44.210 --> 01:11:46.920 Or it could be a cyclic shift thereof. 01:11:46.920 --> 01:11:49.280 So I could, for example, take this part here 01:11:49.280 --> 01:11:51.270 and shift it over to this side. 01:11:51.270 --> 01:11:55.220 So then I'd end up with this, this, and this. 01:11:55.220 --> 01:11:56.606 This is also bitonic. 01:11:56.606 --> 01:11:57.980 or I could shift this whole thing 01:11:57.980 --> 01:12:01.710 and put it over there so I end up with that. 01:12:01.710 --> 01:12:03.470 That's also bitonic. 01:12:03.470 --> 01:12:06.890 So it basically has one max, one min if you're 01:12:06.890 --> 01:12:08.630 reading it cyclically. 01:12:08.630 --> 01:12:10.882 One local max, one local min. 01:12:10.882 --> 01:12:13.090 Except, it's non-decreasing so it's a little messier, 01:12:13.090 --> 01:12:16.230 but same thing. 01:12:16.230 --> 01:12:20.860 OK, let's go over here. 01:12:37.900 --> 01:12:39.870 So if you have a bitonic sequence 01:12:39.870 --> 01:12:43.780 there's something called a bitonic sorting network, which 01:12:43.780 --> 01:12:46.410 you can think of as a parallel sorting algorithm, 01:12:46.410 --> 01:12:47.560 to sort such a thing. 01:12:47.560 --> 01:12:50.700 So it's almost sorted, but not quite. 01:12:50.700 --> 01:12:52.620 And there's this way of sorting them. 01:12:52.620 --> 01:12:57.900 So I'm going to draw a picture to make it easier for me. 01:13:00.692 --> 01:13:02.400 It's a little more annoying to write down 01:13:02.400 --> 01:13:03.450 the general algorithm. 01:13:03.450 --> 01:13:06.260 But so what I want to do, suppose the number of elements 01:13:06.260 --> 01:13:07.014 is a power of 2. 01:13:07.014 --> 01:13:08.180 Here's what I'm going to do. 01:13:08.180 --> 01:13:12.059 First I compare the first and the midway element. 01:13:12.059 --> 01:13:14.100 And then the next one with the next one, and then 01:13:14.100 --> 01:13:16.800 the next one with the next one, the next one with the next one. 01:13:16.800 --> 01:13:18.758 For each of these comparisons I put those items 01:13:18.758 --> 01:13:20.080 in the correct order. 01:13:20.080 --> 01:13:22.250 So if I do that 3 versus 6, 3 is less than 6, 01:13:22.250 --> 01:13:24.800 so 3 stays here, 6 stays there. 01:13:24.800 --> 01:13:27.840 5 is bigger than 4 though, so 4 comes over here, 01:13:27.840 --> 01:13:29.370 5 goes over here. 01:13:29.370 --> 01:13:35.430 Next is 7 versus 2, so 2 comes over here, 7 comes over here. 01:13:35.430 --> 01:13:39.700 Nine versus 0, so 0 is over here, 9 is over there. 01:13:39.700 --> 01:13:42.480 OK, that's an easy set comparisons to do. 01:13:42.480 --> 01:13:45.710 The nice thing is you can do all those in parallel. 01:13:45.710 --> 01:13:47.400 OK, just to check here. 01:13:47.400 --> 01:13:50.460 This was a monotone increasing sub-sequence 01:13:50.460 --> 01:13:53.590 and then a monotone decreasing. 01:13:53.590 --> 01:13:57.270 Now, we have increasing then decreasing. 01:13:57.270 --> 01:14:02.250 And then we have decreasing then increasing. 01:14:02.250 --> 01:14:05.310 But if you look at the left half or the right half, 01:14:05.310 --> 01:14:06.860 they are still bitonic. 01:14:06.860 --> 01:14:08.610 And that's an invariant. 01:14:08.610 --> 01:14:10.560 Bitonic will be preserved here. 01:14:10.560 --> 01:14:12.870 Also notice, all the elements in the left 01:14:12.870 --> 01:14:14.870 are smaller than all the elements in the right. 01:14:14.870 --> 01:14:16.620 That's not true for an arbitrary sequence, 01:14:16.620 --> 01:14:18.328 but it will be true for bitonic sequence. 01:14:18.328 --> 01:14:22.260 Essentially the max is somewhere here 01:14:22.260 --> 01:14:25.050 and so it's going to be spanned by some 01:14:25.050 --> 01:14:27.260 of these intervals, that max. 01:14:27.260 --> 01:14:29.760 And by spanning those intervals, and in particular comparing 01:14:29.760 --> 01:14:31.504 that guy with somebody on the right, 01:14:31.504 --> 01:14:32.920 the max will get over to the right 01:14:32.920 --> 01:14:36.870 and then the rest of the increasing and decreasing 01:14:36.870 --> 01:14:38.530 sub-sequences will follow. 01:14:38.530 --> 01:14:40.440 That's a very hand-wavy argument. 01:14:40.440 --> 01:14:45.000 You can see CLRS for a way to prove that OK, just 01:14:45.000 --> 01:14:46.740 take it for granted now. 01:14:46.740 --> 01:14:48.420 Now, we recurse on the two sides. 01:14:48.420 --> 01:14:52.320 So I compare 3 with 2, 4 with 0, I compare 6 with 7, 01:14:52.320 --> 01:14:53.510 and 6 with 9. 01:14:53.510 --> 01:14:54.990 So what do I get? 01:14:54.990 --> 01:14:56.760 2 is less than 3. 01:14:56.760 --> 01:14:58.570 0 is less than 4. 01:14:58.570 --> 01:15:02.450 Then I compare 6 with 7, so 6 less than 7. 01:15:02.450 --> 01:15:05.910 5 is less than 9. 01:15:05.910 --> 01:15:11.010 So now I've got four chunks, this one, this one, this one, 01:15:11.010 --> 01:15:11.670 and this one. 01:15:11.670 --> 01:15:13.320 I recurse on each of them. 01:15:13.320 --> 01:15:18.110 So 0 is less than 2, 3 is less than 4, 5 is less than 6, 01:15:18.110 --> 01:15:19.660 7 is less than 9. 01:15:19.660 --> 01:15:23.929 Now I have all my items in sorted order, magically. 01:15:23.929 --> 01:15:25.470 OK, but if you believe that invariant 01:15:25.470 --> 01:15:28.680 that the smallest n over two items 01:15:28.680 --> 01:15:30.900 get to the left and the larger n over two items 01:15:30.900 --> 01:15:34.740 get to the right, then by induction this will sort them. 01:15:34.740 --> 01:15:37.431 Kind of like quick sort in that sense. 01:15:37.431 --> 01:15:37.930 OK. 01:15:37.930 --> 01:15:39.600 The cool thing is I can do all these in parallel, 01:15:39.600 --> 01:15:41.474 all these in parallel, all these in parallel. 01:15:41.474 --> 01:15:46.140 So the total time is log k, if I'm a parallel sorting 01:15:46.140 --> 01:15:48.030 algorithm or a sorting network. 01:15:48.030 --> 01:15:51.270 And log k is exactly what I can afford. 01:15:51.270 --> 01:15:54.480 So the last thing to do is to implement an operation like 01:15:54.480 --> 01:15:57.600 this, or like this-- this is kind of the generic picture 01:15:57.600 --> 01:16:00.030 here-- 01:16:00.030 --> 01:16:01.500 in constant time on a word ram. 01:16:13.380 --> 01:16:15.540 One other thing, because I don't want 01:16:15.540 --> 01:16:17.010 to sort a bitonic sequence. 01:16:17.010 --> 01:16:17.890 That wasn't my goal. 01:16:17.890 --> 01:16:20.910 My goal was to merge two sorted sequences. 01:16:20.910 --> 01:16:24.646 I have two sorted sequences, I want to merge them. 01:16:24.646 --> 01:16:25.770 How do I make that bitonic? 01:16:28.460 --> 01:16:32.130 I just flip, like that. 01:16:32.130 --> 01:16:34.642 I take this sequence and I reverse it. 01:16:34.642 --> 01:16:36.600 If I take a sorted sequence and reverse it then 01:16:36.600 --> 01:16:37.391 it looks like this. 01:16:37.391 --> 01:16:39.600 And so then the result is bitonic. 01:16:39.600 --> 01:16:41.850 Then I can use bitonic sorting to sort it. 01:16:41.850 --> 01:16:43.950 So this is the merge problem and I've just 01:16:43.950 --> 01:16:47.220 reduced it to bitonic sorting. 01:16:47.220 --> 01:16:48.060 Cool. 01:16:48.060 --> 01:16:52.610 Except, how do I reverse a sequence 01:16:52.610 --> 01:16:56.770 in log k time on a word ram? 01:16:56.770 --> 01:16:57.480 Cute trick. 01:17:00.735 --> 01:17:04.950 If I have a word and it has many items in it-- 01:17:04.950 --> 01:17:07.042 that's our picture, that's our set up here-- 01:17:07.042 --> 01:17:09.000 and I want to reverse it over all, what I'll do 01:17:09.000 --> 01:17:12.990 is cut it in half, take all these guys put them over here, 01:17:12.990 --> 01:17:15.870 take all these guys put them over here. 01:17:15.870 --> 01:17:18.315 So I've got the right side-- 01:17:18.315 --> 01:17:21.060 sorry, it's only four-- 01:17:21.060 --> 01:17:22.480 cells over here. 01:17:22.480 --> 01:17:24.930 I've got the left side over here. 01:17:24.930 --> 01:17:30.420 Then I recursively reverse the sequence 01:17:30.420 --> 01:17:34.300 and recursively reverse this sequence. 01:17:34.300 --> 01:17:37.220 And then I have the reverse sequence. 01:17:37.220 --> 01:17:39.040 Right, this is a standard. 01:17:39.040 --> 01:17:42.220 One way to reverse items is to do the big part 01:17:42.220 --> 01:17:43.830 and then recursively reverse. 01:17:43.830 --> 01:17:48.430 Then I will get, I mean this is like ab reverse is 01:17:48.430 --> 01:17:51.180 equal to b reverse a reverse. 01:17:51.180 --> 01:17:52.570 Just a fun fact. 01:17:52.570 --> 01:17:55.180 So we're implementing that. 01:17:55.180 --> 01:17:56.810 Now how do I do this? 01:17:56.810 --> 01:17:59.860 Well, this is the thing I've been talking about over here. 01:17:59.860 --> 01:18:02.764 You want to take the high part you just mask that part out, 01:18:02.764 --> 01:18:03.680 shift it to the right. 01:18:03.680 --> 01:18:06.600 You want to take the low part, just mask that part out. 01:18:06.600 --> 01:18:08.440 And so I can take each of these parts 01:18:08.440 --> 01:18:10.330 individually, shift this one to the left, 01:18:10.330 --> 01:18:12.610 shift this one to the right, order them together, 01:18:12.610 --> 01:18:13.690 and I've reversed. 01:18:13.690 --> 01:18:16.000 So this takes constant time. 01:18:16.000 --> 01:18:18.340 This is recursion but the total number of steps 01:18:18.340 --> 01:18:23.770 will be order log k, which is exactly what I can afford. 01:18:23.770 --> 01:18:26.260 So that's how I do the reverse part. 01:18:26.260 --> 01:18:29.620 So if I want to merge, now I have a bitonic sequence. 01:18:29.620 --> 01:18:35.080 Last thing to do is, how do I do this bitonic comparison 01:18:35.080 --> 01:18:36.296 in constant time. 01:18:45.970 --> 01:18:47.665 So, that's next page. 01:18:52.734 --> 01:18:53.650 Here's a quick sketch. 01:19:01.740 --> 01:19:03.690 I'm going to try to draw the generic picture. 01:19:03.690 --> 01:19:05.900 So the generic picture is I have this thing which 01:19:05.900 --> 01:19:08.660 I call a, this thing which I call b, 01:19:08.660 --> 01:19:11.960 and we basically want to compare corresponding items in a and b. 01:19:11.960 --> 01:19:14.330 Though in fact, there's another a over here and another 01:19:14.330 --> 01:19:15.320 b over here. 01:19:15.320 --> 01:19:18.110 In general, there are many a's, many b's, but they appear 01:19:18.110 --> 01:19:19.580 in this nice periodic pattern. 01:19:19.580 --> 01:19:21.650 And so I'm just going to look at one ab pair, 01:19:21.650 --> 01:19:24.500 but it's going to simultaneously work for all of them 01:19:24.500 --> 01:19:26.186 with the same operations. 01:19:26.186 --> 01:19:27.560 So that's the dot, dot, dot here. 01:19:27.560 --> 01:19:29.101 There's another a and then another b, 01:19:29.101 --> 01:19:30.860 and another a and another b. 01:19:30.860 --> 01:19:32.810 So what do I do? 01:19:32.810 --> 01:19:40.760 First thing I do is make 1aa, 1aa, 1aa. 01:19:40.760 --> 01:19:42.510 In other words, I add in these 1 bits. 01:19:42.510 --> 01:19:46.555 Remember, we had zeros there hanging out. 01:19:46.555 --> 01:19:48.230 That was when we were going to use them. 01:19:48.230 --> 01:19:51.030 So I set them all to ones in the a list. 01:19:51.030 --> 01:19:54.095 I take the b list, I just shift it over and leave them 01:19:54.095 --> 01:19:54.595 as zeros. 01:20:00.300 --> 01:20:01.870 Time for bit tricks. 01:20:01.870 --> 01:20:05.160 Then I take this number, I subtract it from this number. 01:20:05.160 --> 01:20:06.610 What do I get? 01:20:06.610 --> 01:20:10.140 I get zeros or ones for whether this bit got 01:20:10.140 --> 01:20:12.030 borrowed, which tells me whether little a is 01:20:12.030 --> 01:20:13.350 bigger than little b. 01:20:13.350 --> 01:20:14.440 Then I get some junk. 01:20:14.440 --> 01:20:15.300 And then I get 0, 1. 01:20:15.300 --> 01:20:16.320 Junk. 01:20:16.320 --> 01:20:17.310 So on. 01:20:17.310 --> 01:20:20.730 0, 1, junk. 01:20:20.730 --> 01:20:23.880 So 0 corresponds to a being smaller 01:20:23.880 --> 01:20:26.720 than b, and 1 corresponds to the reverse. 01:20:26.720 --> 01:20:27.800 OK, whatever. 01:20:27.800 --> 01:20:31.820 Then I do my usual masking, just the 0, 1 bits, 01:20:31.820 --> 01:20:32.820 then there's some zeros. 01:20:36.640 --> 01:20:38.760 OK, here's a new trick which we haven't seen yet. 01:20:38.760 --> 01:20:42.450 I'm going to shift this to the right by two, or in general 01:20:42.450 --> 01:20:44.470 by the size of those items. 01:20:44.470 --> 01:20:48.810 So now I have 0, 1 here, 0, 1 here, 0, 1 here. 01:20:48.810 --> 01:20:50.880 Each of these bits might be different. 01:20:50.880 --> 01:20:53.140 So I take this, shift to the right. 01:20:53.140 --> 01:20:55.560 Now I subtract. 01:20:55.560 --> 01:20:57.400 What this gives me-- 01:20:57.400 --> 01:21:00.690 so I'm taking 1, 0, 0, 0, minus 1. 01:21:00.690 --> 01:21:03.750 What I get is 0, 1, 1, 1. 01:21:03.750 --> 01:21:06.160 OK, in this case, there's just two bits. 01:21:06.160 --> 01:21:07.230 Except it's not 1, 1. 01:21:07.230 --> 01:21:14.760 It's either 0, 0 or 1, 1. 01:21:14.760 --> 01:21:22.120 And then we have here 0, and then 0, 0 or 1, 1, and so on. 01:21:22.120 --> 01:21:24.630 So now these are masks. 01:21:24.630 --> 01:21:28.410 The zeros correspond to when the a's were smaller, 01:21:28.410 --> 01:21:31.020 the ones correspond to when the b's are smaller. 01:21:31.020 --> 01:21:38.430 So if I take this [? added ?] with this, I get the large b's. 01:21:38.430 --> 01:21:42.480 If I take this [? added ?] with this, I get the small a's. 01:21:42.480 --> 01:21:45.630 Or one of some of those combinations. 01:21:45.630 --> 01:21:48.240 The point is I take-- 01:21:48.240 --> 01:21:51.895 I think here I get the small a's. 01:21:55.080 --> 01:22:05.265 And then if I take this thing and negate it and then mask it 01:22:05.265 --> 01:22:07.740 with this I get the small b's. 01:22:11.750 --> 01:22:13.120 And then I take this thing-- 01:22:13.120 --> 01:22:14.950 I ran out of room. 01:22:14.950 --> 01:22:19.210 So I've got small a's, small b's. 01:22:22.240 --> 01:22:25.450 I take this thing I shift it over so I have small b's here. 01:22:29.020 --> 01:22:33.620 And then I OR these together, I get all the smalls. 01:22:33.620 --> 01:22:36.940 OK, these are designed to never conflict because I 01:22:36.940 --> 01:22:38.604 took the inversion here. 01:22:38.604 --> 01:22:40.770 So I get the smaller of the a's, smaller of the b's. 01:22:40.770 --> 01:22:44.590 So now I put the smaller things in the right order. 01:22:44.590 --> 01:22:48.170 I do exactly the same procedure negated, and get the larges. 01:22:51.210 --> 01:22:54.530 And then I OR these together and I get sorted. 01:22:57.430 --> 01:22:58.330 Well, not sorted. 01:22:58.330 --> 01:23:01.600 I get whatever bitonic sort would have done. 01:23:01.600 --> 01:23:05.140 It's doing these pairwise comparisons by doing this shift 01:23:05.140 --> 01:23:06.970 and it's putting things in the right order. 01:23:06.970 --> 01:23:08.890 The small ones always end up on the left, 01:23:08.890 --> 01:23:10.640 the large ones always end up on the right. 01:23:10.640 --> 01:23:13.790 That's the definition of the operation we want to do. 01:23:13.790 --> 01:23:16.769 Bitonic operated. 01:23:16.769 --> 01:23:19.060 And because we are just doing shifts, and subtractions, 01:23:19.060 --> 01:23:21.610 and all these things, it works even though there's not just 01:23:21.610 --> 01:23:23.830 one ab pair, but there's a whole bunch of ab pairs. 01:23:23.830 --> 01:23:28.460 They will all be shifted, and computed, and sorted together. 01:23:28.460 --> 01:23:31.210 It's a little hard to visualize but this 01:23:31.210 --> 01:23:34.090 will do one bitonic operation and a constant number 01:23:34.090 --> 01:23:35.890 of word ram operations. 01:23:35.890 --> 01:23:38.350 And so you pop out all these stacks 01:23:38.350 --> 01:23:41.110 this gives us a merger, that gives us packed sorting, 01:23:41.110 --> 01:23:45.130 and with packed sorting we get signature sorting. 01:23:45.130 --> 01:23:47.351 Easy, right? 01:23:47.351 --> 01:23:49.600 All right, that's probably one of the most complicated 01:23:49.600 --> 01:23:51.058 algorithms we'll see in this class. 01:23:51.058 --> 01:23:53.230 But from that you get a priority queue that 01:23:53.230 --> 01:23:56.110 runs in constant time provided w is a little bigger than log 01:23:56.110 --> 01:23:57.010 squared n. 01:23:57.010 --> 01:23:58.560 Boom.