WEBVTT 00:00:06.000 --> 00:00:10.000 And finally we look at the 6 And at that point we are done. 00:00:10.000 --> 00:00:11.665 We're going to get started. 00:00:11.665 --> 00:00:16.000 Handouts are the by the door if anybody didn't pick one up. 00:00:16.000 --> 00:00:18.000 My name is Charles Leiserson. 00:00:18.000 --> 00:00:22.500 I will be lecturing this course this term, Introduction 00:00:22.500 --> 00:00:25.000 to Algorithms, with Erik Demaine. 00:00:25.000 --> 00:00:29.200 In addition, this is an SMA course, a Singapore MIT 00:00:29.200 --> 00:00:35.000 Alliance course which will be run in Singapore by David Hsu. 00:00:35.000 --> 00:00:39.160 And so all the lectures will be videotaped and made 00:00:39.160 --> 00:00:42.724 available on the Web for the Singapore students, 00:00:42.724 --> 00:00:47.664 as well as for MIT students who choose to watch them 00:00:47.664 --> 00:00:49.000 on the Web. 00:00:49.000 --> 00:00:55.000 If you have an issue of not wanting to be on the videotape, 00:00:55.000 --> 00:00:57.500 you should sit in the back row. 00:00:57.500 --> 00:00:58.000 OK? 00:00:58.000 --> 00:01:00.724 Otherwise, you will be on it. 00:01:00.724 --> 00:01:03.375 There is a video recording policy, 00:01:03.375 --> 00:01:06.000 but it seems like they ran out. 00:01:06.000 --> 00:01:08.331 If anybody wants to see it, people, 00:01:08.331 --> 00:01:11.089 if they could just sort of pass them 00:01:11.089 --> 00:01:14.666 around maybe a little bit, once you're done reading it, 00:01:14.666 --> 00:01:16.331 or you can come up. 00:01:16.331 --> 00:01:18.000 I did secure one copy. 00:01:18.000 --> 00:01:21.000 Before we get into the content of the course, 00:01:21.000 --> 00:01:24.108 let's briefly go over the course information 00:01:24.108 --> 00:01:27.724 because there are some administrative things that we 00:01:27.724 --> 00:01:30.000 sort of have to do. 00:01:30.000 --> 00:01:33.000 As you can see, this term we have a big staff. 00:01:33.000 --> 00:01:35.000 Take a look at the handout here. 00:01:35.000 --> 00:01:37.496 Including this term six TAs, which 00:01:37.496 --> 00:01:42.500 is two more TAs than we normally get for this course. 00:01:42.500 --> 00:01:45.666 That means recitations will be particularly small. 00:01:45.666 --> 00:01:50.500 There is a World Wide Web page, and you should bookmark that 00:01:50.500 --> 00:01:55.142 and go there regularly because that is where 00:01:55.142 --> 00:01:57.834 everything will be distributed. 00:01:57.834 --> 00:01:58.333 Email. 00:01:58.333 --> 00:02:00.800 You should not be emailing directly to, 00:02:00.800 --> 00:02:04.000 even though we give you our email addresses, 00:02:04.000 --> 00:02:06.000 to the individual members of the staff. 00:02:06.000 --> 00:02:07.665 You should email us generally. 00:02:07.665 --> 00:02:11.284 And the reason is you will get much faster response. 00:02:11.284 --> 00:02:13.500 And also, for any communications, 00:02:13.500 --> 00:02:17.600 generally we like to monitor what the communications are 00:02:17.600 --> 00:02:21.284 so it's helpful to have emails coming to everybody 00:02:21.284 --> 00:02:23.000 on the course staff. 00:02:23.000 --> 00:02:26.999 As I mentioned, we will be doing distance learning this term. 00:02:26.999 --> 00:02:29.500 And so you can watch lectures online 00:02:29.500 --> 00:02:32.500 if you choose to do that. 00:02:32.500 --> 00:02:37.000 I would recommend, for people who have the opportunity 00:02:37.000 --> 00:02:38.999 to watch, to come live. 00:02:38.999 --> 00:02:40.000 It's better live. 00:02:40.000 --> 00:02:42.000 You get to interact. 00:02:42.000 --> 00:02:45.815 There's an intangible that comes with having it live. 00:02:45.815 --> 00:02:48.454 In fact, in addition to the videos, 00:02:48.454 --> 00:02:51.632 I meet weekly with the Singapore students 00:02:51.632 --> 00:02:57.284 so that they have a live session as well. 00:02:57.284 --> 00:02:58.000 Prerequisites. 00:02:58.000 --> 00:03:00.140 The prerequisites for this course 00:03:00.140 --> 00:03:05.000 are 6.042, which is Math for Computer Science, and 6.001. 00:03:05.000 --> 00:03:09.000 You basically need discrete mathematics and probability, 00:03:09.000 --> 00:03:11.220 as well as programming experience 00:03:11.220 --> 00:03:13.500 to take this course successfully. 00:03:13.500 --> 00:03:17.284 People do not have that background should not 00:03:17.284 --> 00:03:19.000 be in the class. 00:03:19.000 --> 00:03:22.000 We will be checking prerequisites. 00:03:22.000 --> 00:03:24.496 If you have any questions, please 00:03:24.496 --> 00:03:27.666 come to talk to us after class. 00:03:27.666 --> 00:03:29.000 Let's see. 00:03:29.000 --> 00:03:30.500 Lectures are here. 00:03:30.500 --> 00:03:33.452 For SMA students, they have the videotapes 00:03:33.452 --> 00:03:36.428 and they will also have a weekly meeting. 00:03:36.428 --> 00:03:41.220 Students must attend a one hour recitation session each week. 00:03:41.220 --> 00:03:45.712 There will be new material presented in the recitation. 00:03:45.712 --> 00:03:49.500 Unlike the lectures, they will not be online. 00:03:49.500 --> 00:03:52.200 Unlike the lectures, there will not 00:03:52.200 --> 00:03:56.142 be lecture notes distributed for the recitations in general. 00:03:56.142 --> 00:04:00.000 And, yet, there will be material there 00:04:00.000 --> 00:04:03.000 that is directly on the exams. 00:04:03.000 --> 00:04:07.000 And so every term we say oh, when did you cover that? 00:04:07.000 --> 00:04:08.500 That was in recitation. 00:04:08.500 --> 00:04:10.000 You missed that one. 00:04:10.000 --> 00:04:12.284 So, recitations are mandatory. 00:04:12.284 --> 00:04:15.452 And, in particular, also let me just 00:04:15.452 --> 00:04:18.416 mention your recitation instructor is the one who 00:04:18.416 --> 00:04:20.080 assigns your final grade. 00:04:20.080 --> 00:04:24.500 So we have a grade meeting and keep everybody normal, 00:04:24.500 --> 00:04:29.500 but your recitation has the final say on your grade. 00:04:29.500 --> 00:04:30.000 Handouts. 00:04:30.000 --> 00:04:34.000 Handouts are available on the course Web page. 00:04:34.000 --> 00:04:39.142 We will not generally, except for this one, first handout, 00:04:39.142 --> 00:04:42.000 be bringing handouts to class. 00:04:42.000 --> 00:04:46.000 Textbook is this book, Introduction to Algorithms. 00:04:46.000 --> 00:04:50.500 MIT students can get it any of the local bookstores, including 00:04:50.500 --> 00:04:52.000 the MIT Coop. 00:04:52.000 --> 00:04:55.545 There is also a new online service 00:04:55.545 --> 00:04:57.180 that provides textbooks. 00:04:57.180 --> 00:05:02.125 You can also get a discount if you buy it 00:05:02.125 --> 00:05:04.000 at the MIT Press Bookstore. 00:05:04.000 --> 00:05:09.450 There is a coupon in the MIT Student Telephone Directory 00:05:09.450 --> 00:05:11.998 for a discount on MIT Press books. 00:05:11.998 --> 00:05:17.000 And you can use that to purchase this book at a discount. 00:05:17.000 --> 00:05:18.142 Course website. 00:05:18.142 --> 00:05:21.000 This is the course website. 00:05:21.000 --> 00:05:24.108 It links to the Stellar website, which 00:05:24.108 --> 00:05:30.000 is where, actually, everything will be kept. 00:05:30.000 --> 00:05:33.000 And SMA students have their own website. 00:05:33.000 --> 00:05:36.108 Some students find this course particularly challenges 00:05:36.108 --> 00:05:38.776 so we will have extra help. 00:05:38.776 --> 00:05:42.500 We will post weekly office hours on the course 00:05:42.500 --> 00:05:44.000 website for the TAs. 00:05:44.000 --> 00:05:46.912 And then as an experiment this term, 00:05:46.912 --> 00:05:51.000 we are going to offer homework labs for this class. 00:05:51.000 --> 00:05:55.331 What a homework lab is, is it's a place and a time 00:05:55.331 --> 00:05:58.500 you can go where other people in the course 00:05:58.500 --> 00:06:01.000 will go to do homework. 00:06:01.000 --> 00:06:05.360 And there will be typically two TAs who staff the lab. 00:06:05.360 --> 00:06:07.666 And so, as you're working on your homework, 00:06:07.666 --> 00:06:11.332 you can get help from the TAs if you need it. 00:06:11.332 --> 00:06:14.665 And it's generally a place, we're going to schedule those, 00:06:14.665 --> 00:06:18.535 and they will be on the course calendar for where it is 00:06:18.535 --> 00:06:22.152 and when it is that they will be held, but usually 00:06:22.152 --> 00:06:26.444 Sundays 2:00 to 4:00 pm, or else it will be some evening. 00:06:26.444 --> 00:06:29.250 I think the first one is an evening, right? 00:06:29.250 --> 00:06:33.333 Near to when the homework is due. 00:06:33.333 --> 00:06:36.428 Your best bet is try to do the homework 00:06:36.428 --> 00:06:39.000 in advance of the homework lab. 00:06:39.000 --> 00:06:41.149 But then, if you want extra help, 00:06:41.149 --> 00:06:45.000 if you want to talk over your solutions with people 00:06:45.000 --> 00:06:49.000 because as we will talk about problem sets 00:06:49.000 --> 00:06:54.200 you can solve in collaboration with other people in the class. 00:06:54.200 --> 00:07:00.000 In addition, there are several peer assistance programs. 00:07:00.000 --> 00:07:02.664 Also the office of Minority Education 00:07:02.664 --> 00:07:05.776 has an assistance program, and those usually 00:07:05.776 --> 00:07:08.000 get booked up pretty quickly. 00:07:08.000 --> 00:07:10.800 If you're interested in those, good idea 00:07:10.800 --> 00:07:15.000 to make an appointment to get there and get help soon. 00:07:15.000 --> 00:07:19.666 The homework labs, I hope a lot of people will try that out. 00:07:19.666 --> 00:07:21.000 We've never done this. 00:07:21.000 --> 00:07:24.000 I don't know of any other course. 00:07:24.000 --> 00:07:28.000 Do other people know of courses at MIT that have done this? 00:07:28.000 --> 00:07:30.928 6.011 did it, OK. 00:07:30.928 --> 00:07:31.428 Good. 00:07:31.428 --> 00:07:34.500 And was it successful in that class? 00:07:34.500 --> 00:07:35.962 It never went, 00:07:35.962 --> 00:07:36.461 OK. 00:07:36.461 --> 00:07:38.766 Good. [LAUGHTER] We will see. 00:07:38.766 --> 00:07:43.000 If it's not paying off then we will just 00:07:43.000 --> 00:07:47.000 return to ordinary office hours for those TAs, 00:07:47.000 --> 00:07:52.000 but I think for some students that is a good opportunity. 00:07:52.000 --> 00:07:55.852 If you wish to be registered in this course, 00:07:55.852 --> 00:08:00.664 you must sign up on the course Web page. 00:08:00.664 --> 00:08:04.000 So, that is requirement one. 00:08:04.000 --> 00:08:06.270 It must be done today. 00:08:06.270 --> 00:08:11.800 You will find it difficult to pass the course if you are not 00:08:11.800 --> 00:08:13.000 in the class. 00:08:13.000 --> 00:08:16.424 And you should notify your TA if you 00:08:16.424 --> 00:08:20.452 decide to drop so that we can get you off 00:08:20.452 --> 00:08:23.000 and stop the mailings, stop the spam. 00:08:23.000 --> 00:08:29.000 And you should register today before 7:00 PM. 00:08:29.000 --> 00:08:32.600 And then we're going to email your recitation assignment 00:08:32.600 --> 00:08:34.200 to you before Noon tomorrow. 00:08:34.200 --> 00:08:37.816 And if you don't receive this information by Thursday Noon, 00:08:37.816 --> 00:08:41.856 please send us an email to the course staff 00:08:41.856 --> 00:08:44.000 generally, not to me individually, 00:08:44.000 --> 00:08:48.000 saying that you didn't receive your recitation assignment. 00:08:48.000 --> 00:08:51.070 And so if you haven't received it by Thursday Noon 00:08:51.070 --> 00:08:52.000 you want to. 00:08:52.000 --> 00:08:55.267 I think generally they are going to send them 00:08:55.267 --> 00:08:59.426 out tonight or at least by tomorrow morning. 00:08:59.426 --> 00:09:00.000 Yeah. 00:09:00.000 --> 00:09:00.499 OK. 00:09:00.499 --> 00:09:02.459 SMA students don't have to worry about this. 00:09:02.459 --> 00:09:03.000 Problem sets. 00:09:03.000 --> 00:09:06.630 We have nine problem sets that we project will be 00:09:06.630 --> 00:09:08.125 assigned during the semester. 00:09:08.125 --> 00:09:10.400 A couple things about problem sets. 00:09:10.400 --> 00:09:12.428 Homeworks won't generally be accepted, 00:09:12.428 --> 00:09:15.000 if you have extenuating circumstances you 00:09:15.000 --> 00:09:17.500 should make prior arrangements with your recitation 00:09:17.500 --> 00:09:18.000 instructor. 00:09:18.000 --> 00:09:21.000 In fact, almost all of the administrative stuff, 00:09:21.000 --> 00:09:23.565 you shouldn't come to me to ask and say 00:09:23.565 --> 00:09:25.285 can I hand in something late? 00:09:25.285 --> 00:09:28.250 You should be talking to your recitation instructor. 00:09:28.250 --> 00:09:32.999 You can read the other things about the form, 00:09:32.999 --> 00:09:36.664 but let me just mention that there are exercises that 00:09:36.664 --> 00:09:41.875 should be solved but not handed in as well to give you drill 00:09:41.875 --> 00:09:43.000 on the material. 00:09:43.000 --> 00:09:46.000 I highly recommend you doing the exercises. 00:09:46.000 --> 00:09:50.000 They both test your understanding of the material, 00:09:50.000 --> 00:09:55.000 and exercises have this way of finding themselves on quizzes. 00:09:55.000 --> 00:10:00.000 You're often asked to describe algorithms. 00:10:00.000 --> 00:10:04.280 And here is a little outline of what you can 00:10:04.280 --> 00:10:06.500 use to describe an algorithm. 00:10:06.500 --> 00:10:11.400 The grading policy is something that somehow I cover. 00:10:11.400 --> 00:10:15.832 And always every term there are at least 00:10:15.832 --> 00:10:20.888 a couple of students who pretend like I never showed them this. 00:10:20.888 --> 00:10:28.285 If you skip problems it has a nonlinear effect on your grade. 00:10:28.285 --> 00:10:30.000 Nonlinear, OK? 00:10:30.000 --> 00:10:34.000 If you don't skip any problems, no effect on your grade. 00:10:34.000 --> 00:10:38.000 If you skip one problem, a hundredth of a letter grade, 00:10:38.000 --> 00:10:39.600 we can handle that. 00:10:39.600 --> 00:10:42.000 But two problems it's a tenth. 00:10:42.000 --> 00:10:47.332 And, as you see, by the time you have skipped like five letter 00:10:47.332 --> 00:10:50.000 grades, it is already five problems. 00:10:50.000 --> 00:10:53.000 This is not problem sets, by the way. 00:10:53.000 --> 00:10:54.000 This is problems, OK? 00:10:54.000 --> 00:10:59.000 You're down a third of a letter grade. 00:10:59.000 --> 00:11:01.664 And if you don't do nine or more, 00:11:01.664 --> 00:11:05.000 so that's typically about three to four problem sets, 00:11:05.000 --> 00:11:07.000 you don't pass the class. 00:11:07.000 --> 00:11:11.000 I always have some students coming at the end of the year 00:11:11.000 --> 00:11:14.000 saying oh, I didn't do any of my problems. 00:11:14.000 --> 00:11:18.000 Can you just pass me because I did OK on the exams? 00:11:18.000 --> 00:11:23.000 Answer no, a very simple answer because we've said it upfront. 00:11:23.000 --> 00:11:27.000 So, the problem sets are an integral part of the course. 00:11:27.000 --> 00:11:28.428 Collaboration policy. 00:11:28.428 --> 00:11:32.900 This is extremely important so everybody pay attention. 00:11:32.900 --> 00:11:35.000 If you are asleep now wake up. 00:11:35.000 --> 00:11:39.000 Like that's going to wake anybody up, right? 00:11:39.000 --> 00:11:41.000 [LAUGHTER] The goal of homework. 00:11:41.000 --> 00:11:43.000 Professor Demaine and my philosophy 00:11:43.000 --> 00:11:48.000 is that the goal of homework is to help you learn the material. 00:11:48.000 --> 00:11:50.565 And one way of helping to learn is not 00:11:50.565 --> 00:11:53.600 to just be stuck and unable to solve something 00:11:53.600 --> 00:11:56.400 because then you're in no better shape 00:11:56.400 --> 00:12:00.571 when the exam roles around, which is where we're actually 00:12:00.571 --> 00:12:01.713 evaluating you. 00:12:01.713 --> 00:12:04.500 So, you're encouraged to collaborate. 00:12:04.500 --> 00:12:08.332 But there are some commonsense things about collaboration. 00:12:08.332 --> 00:12:12.089 If you go and you collaborate to the extent 00:12:12.089 --> 00:12:15.856 that all you're doing is getting the information from somebody 00:12:15.856 --> 00:12:18.363 else, you're not learning the material 00:12:18.363 --> 00:12:22.000 and you're not going to do well on the exams. 00:12:22.000 --> 00:12:25.625 In our experience, students who collaborate generally 00:12:25.625 --> 00:12:30.000 do better than students who work alone. 00:12:30.000 --> 00:12:31.840 But you owe it to yourself, if you're 00:12:31.840 --> 00:12:36.200 going to work in a study group, to be prepared for your study 00:12:36.200 --> 00:12:37.000 group meeting. 00:12:37.000 --> 00:12:39.448 And specifically you should spend a half an hour 00:12:39.448 --> 00:12:41.842 to 45 minutes on each problem before you 00:12:41.842 --> 00:12:44.428 go to group so you're up to speed 00:12:44.428 --> 00:12:47.000 and you've tried out your ideas. 00:12:47.000 --> 00:12:48.610 And you may have solutions to some, 00:12:48.610 --> 00:12:50.800 you may be stuck on some other ones, 00:12:50.800 --> 00:12:54.000 but at least you applied yourself to it. 00:12:54.000 --> 00:12:57.000 After 30 to 45 minutes, if you cannot get the problem, 00:12:57.000 --> 00:13:00.928 just sitting there and banging your head against it makes no 00:13:00.928 --> 00:13:01.428 sense. 00:13:01.428 --> 00:13:04.666 It's not a productive use of your time. 00:13:04.666 --> 00:13:08.000 And I know most of you have issues with having time 00:13:08.000 --> 00:13:09.000 on your hands, right? 00:13:09.000 --> 00:13:10.600 Like it's not there. 00:13:10.600 --> 00:13:13.600 So, don't go banging your head against problems 00:13:13.600 --> 00:13:16.333 that are too hard or where you don't understand 00:13:16.333 --> 00:13:18.000 what's going on or whatever. 00:13:18.000 --> 00:13:21.000 That's when the study group can help out. 00:13:21.000 --> 00:13:23.664 And, as I mentioned, we'll have homework labs 00:13:23.664 --> 00:13:25.665 which will give you an opportunity 00:13:25.665 --> 00:13:29.200 to go and do that and coordinate with other students 00:13:29.200 --> 00:13:32.750 rather than necessarily having to form your own group. 00:13:32.750 --> 00:13:35.000 And the TAs will be there. 00:13:35.000 --> 00:13:39.160 If your group is unable to solve the problem then 00:13:39.160 --> 00:13:43.000 talk to other groups or ask your recitation instruction. 00:13:43.000 --> 00:13:46.000 And, that's how you go about solving them. 00:13:46.000 --> 00:13:49.330 Writing up the problem sets, however, 00:13:49.330 --> 00:13:51.500 is your individual responsibility 00:13:51.500 --> 00:13:54.000 and should be done alone. 00:13:54.000 --> 00:13:58.000 You don't write up your problem solutions with other students, 00:13:58.000 --> 00:14:01.227 you write them up on your own. 00:14:01.227 --> 00:14:04.428 And you should on your problem sets, 00:14:04.428 --> 00:14:07.000 because this is an academic place, 00:14:07.000 --> 00:14:11.000 we understand that the source of academic information 00:14:11.000 --> 00:14:15.000 is very important, if you collaborated on solutions 00:14:15.000 --> 00:14:18.664 you should write a list of the collaborators. 00:14:18.664 --> 00:14:22.750 Say I worked with so and so on this solution. 00:14:22.750 --> 00:14:25.000 It does not affect your grade. 00:14:25.000 --> 00:14:30.000 It's just a question of being scholarly. 00:14:30.000 --> 00:14:34.363 It is a violation of this policy to submit a problem solution 00:14:34.363 --> 00:14:38.500 that you cannot orally explain to a member of the course 00:14:38.500 --> 00:14:39.000 staff. 00:14:39.000 --> 00:14:44.000 You say oh, well, my write up is similar to that other person's. 00:14:44.000 --> 00:14:45.600 I didn't copy them. 00:14:45.600 --> 00:14:49.284 We may ask you to orally explain your solution. 00:14:49.284 --> 00:14:52.776 If you are unable, according to this policy, 00:14:52.776 --> 00:14:55.375 the presumption is that you cheated. 00:14:55.375 --> 00:14:59.800 So, do not write up stuff that you don't understand. 00:14:59.800 --> 00:15:06.220 You should be able to write up the stuff that you understand. 00:15:06.220 --> 00:15:10.220 Understand why you're putting down what you're putting down. 00:15:10.220 --> 00:15:13.284 If it isn't obvious, no collaboration whatsoever 00:15:13.284 --> 00:15:15.000 is permitted on exams. 00:15:15.000 --> 00:15:17.496 Exams is when we evaluate you. 00:15:17.496 --> 00:15:21.284 And now we're not interested in evaluating other people, 00:15:21.284 --> 00:15:23.500 we're interested in evaluating you. 00:15:23.500 --> 00:15:26.000 So, no collaboration on exams. 00:15:26.000 --> 00:15:31.000 We will have a take home exam for the second quiz. 00:15:31.000 --> 00:15:33.000 You should look at the schedule. 00:15:33.000 --> 00:15:36.000 If there are problems with the schedule of that, 00:15:36.000 --> 00:15:37.360 we want to know early. 00:15:37.360 --> 00:15:39.444 And we will give you more details 00:15:39.444 --> 00:15:43.500 about the collaboration in the lecture on Monday, November 00:15:43.500 --> 00:15:44.000 28th. 00:15:44.000 --> 00:15:47.500 Now, generally, the lectures here, they're mandatory 00:15:47.500 --> 00:15:52.272 and you have to know them, but I know that some people say gee, 00:15:52.272 --> 00:15:53.904 9:30 is kind of early, especially 00:15:53.904 --> 00:15:55.333 on a Monday or whatever. 00:15:55.333 --> 00:15:58.750 It can be kind of early to get up. 00:15:58.750 --> 00:16:01.800 However, on Monday, November 28th, 00:16:01.800 --> 00:16:07.304 you fail the exam if you do not show up to lecture on time. 00:16:07.304 --> 00:16:10.000 That one day you must show up. 00:16:10.000 --> 00:16:11.452 Any questions about that? 00:16:11.452 --> 00:16:14.400 That one day you must show up here, 00:16:14.400 --> 00:16:18.000 even if you've been watching them on the Web. 00:16:18.000 --> 00:16:21.333 And generally, if you think you have transgressed, 00:16:21.333 --> 00:16:25.500 the best is to come to us to talk about it. 00:16:25.500 --> 00:16:28.571 We can usually work something out. 00:16:28.571 --> 00:16:34.270 It's when we find somebody has transgressed from a third party 00:16:34.270 --> 00:16:38.500 or from obvious analyses that we do with homeworks, 00:16:38.500 --> 00:16:41.000 that's when things get messy. 00:16:41.000 --> 00:16:45.000 So, if you think, for some reason or other, 00:16:45.000 --> 00:16:47.331 oh, I may have done something wrong, 00:16:47.331 --> 00:16:49.500 please come and talk to us. 00:16:49.500 --> 00:16:54.855 We actually were students once, too, albeit many years ago. 00:16:54.855 --> 00:16:56.000 Any questions? 00:16:56.000 --> 00:17:00.000 So, this class has great material. 00:17:00.000 --> 00:17:02.000 Fabulous material. 00:17:02.000 --> 00:17:13.000 And it's really fun, but you do have to work hard. 00:17:13.000 --> 00:17:16.000 Let's talk content. 00:17:29.000 --> 00:17:32.000 This is the topic of the first part of the course. 00:17:32.000 --> 00:17:35.000 The first part of the course is focused on analysis. 00:17:35.000 --> 00:17:39.000 The second part of the course is focused on design. 00:17:39.000 --> 00:17:41.331 Before you can do design, you have 00:17:41.331 --> 00:17:45.000 to master a bunch of techniques for analyzing algorithms. 00:17:45.000 --> 00:17:49.000 And then you'll be in a position to design algorithms 00:17:49.000 --> 00:17:52.000 that you can analyze and that which are efficient. 00:17:52.000 --> 00:18:06.333 The analysis of algorithm is the theoretical study -- -- 00:18:06.333 --> 00:18:21.220 of computer program performance -- -- and resource usage. 00:18:21.220 --> 00:18:24.666 And a particular focus on performance. 00:18:24.666 --> 00:18:29.500 We're studying how to make things fast. 00:18:29.500 --> 00:18:32.250 In particular, computer programs. 00:18:32.250 --> 00:18:37.998 We also will discover and talk about other resources 00:18:37.998 --> 00:18:43.000 such as communication, such as memory, whether RAM memory 00:18:43.000 --> 00:18:44.666 or disk memory. 00:18:44.666 --> 00:18:49.776 There are other resources that we may care about, 00:18:49.776 --> 00:18:52.714 but predominantly we focus on performance. 00:18:52.714 --> 00:18:57.500 Because this is a course about performance, 00:18:57.500 --> 00:19:02.625 I like to put things in perspective a little bit 00:19:02.625 --> 00:19:10.000 by starting out and asking, in programming, what is more 00:19:10.000 --> 00:19:13.000 important than performance? 00:19:13.000 --> 00:19:18.000 If you're in an engineering situation and writing code, 00:19:18.000 --> 00:19:24.500 writing software, what's more important than performance? 00:19:24.500 --> 00:19:26.000 Correctness. 00:19:26.000 --> 00:19:26.714 Good. 00:19:26.714 --> 00:19:27.428 OK. 00:19:27.428 --> 00:19:28.856 What else? 00:19:28.856 --> 00:19:31.000 Simplicity can be. 00:19:31.000 --> 00:19:31.501 Very good. 00:19:31.501 --> 00:19:32.000 Yeah. 00:19:32.000 --> 00:19:40.000 Maintainability often much more important than performance. 00:19:40.000 --> 00:19:40.500 Cost. 00:19:40.500 --> 00:19:44.714 And what type of cost are you thinking? 00:19:44.714 --> 00:19:49.000 No, I mean cost of what? 00:19:49.000 --> 00:19:53.000 We're talking software here, right? 00:19:53.000 --> 00:20:00.000 What type of cost do you have in mind? 00:20:00.000 --> 00:20:04.500 There are some costs that are involved when programming 00:20:04.500 --> 00:20:05.856 like programmer time. 00:20:05.856 --> 00:20:10.500 So, programmer time is another thing also that might be. 00:20:10.500 --> 00:20:11.000 Stability. 00:20:11.000 --> 00:20:13.000 Robustness of the software. 00:20:13.000 --> 00:20:16.000 Does it break all the time? 00:20:16.000 --> 00:20:18.000 What else? 00:20:25.000 --> 00:20:25.750 Come on. 00:20:25.750 --> 00:20:28.400 We've got a bunch of engineers here. 00:20:28.400 --> 00:20:30.000 A lot of things. 00:20:30.000 --> 00:20:31.125 How about features? 00:20:31.125 --> 00:20:33.000 Features can be more important. 00:20:33.000 --> 00:20:37.000 Having a wider collection of features than your competitors. 00:20:37.000 --> 00:20:38.000 Functionality. 00:20:38.000 --> 00:20:39.000 Modularity. 00:20:39.000 --> 00:20:42.927 Is it designed in a way where you can make changes 00:20:42.927 --> 00:20:47.377 in a local part of the code and you don't have to make changes 00:20:47.377 --> 00:20:50.904 across the code in order to affect a simple change 00:20:50.904 --> 00:20:52.000 in the functionality? 00:20:52.000 --> 00:20:54.800 There is one big one which definitely, 00:20:54.800 --> 00:21:01.000 especially in the `90s, was like the big thing in computers. 00:21:01.000 --> 00:21:01.999 The big thing. 00:21:01.999 --> 00:21:03.000 Well, security actually. 00:21:03.000 --> 00:21:03.500 Good. 00:21:03.500 --> 00:21:06.500 I don't even have that one down. 00:21:06.500 --> 00:21:08.000 Security is excellent. 00:21:08.000 --> 00:21:11.000 That's actually been more in the 2000. 00:21:11.000 --> 00:21:13.250 Security has been far more important 00:21:13.250 --> 00:21:14.600 often than performance. 00:21:14.600 --> 00:21:18.200 Scalability has been important, although scalability, 00:21:18.200 --> 00:21:21.500 in some sense, is performance related. 00:21:21.500 --> 00:21:24.000 But, yes, scalability is good. 00:21:24.000 --> 00:21:29.000 What was the big breakthrough and why do people use Macintosh 00:21:29.000 --> 00:21:30.998 rather than Windows, those people who 00:21:30.998 --> 00:21:34.000 are of that religion? 00:21:34.000 --> 00:21:35.750 User-friendliness. 00:21:35.750 --> 00:21:36.250 Wow. 00:21:36.250 --> 00:21:40.500 If you look at the number of cycles of computers that went 00:21:40.500 --> 00:21:43.363 into user friendliness in the `90s, 00:21:43.363 --> 00:21:48.110 it grew from almost nothing to where it's now the vast part 00:21:48.110 --> 00:21:52.000 of the computation goes into user friendly. 00:21:52.000 --> 00:21:56.000 So, all those things are more important than performance. 00:21:56.000 --> 00:22:00.000 This is a course on performance. 00:22:00.000 --> 00:22:04.160 Then you can say OK, well, why do we bother 00:22:04.160 --> 00:22:08.816 and why study algorithms and performance if it's 00:22:08.816 --> 00:22:12.600 at the bottom of the heap? 00:22:12.600 --> 00:22:15.714 Almost always people would rather 00:22:15.714 --> 00:22:20.000 have these other things than performance. 00:22:20.000 --> 00:22:24.000 You go off and you say to somebody, 00:22:24.000 --> 00:22:32.000 would I rather have performance or more user friendliness? 00:22:32.000 --> 00:22:36.000 It's almost always more important than performance. 00:22:36.000 --> 00:22:38.500 Why do we care then? 00:22:38.500 --> 00:22:39.000 Yeah? 00:22:44.000 --> 00:22:45.712 That wasn't user friendly. 00:22:45.712 --> 00:22:49.250 Sometimes performance is correlated with user 00:22:49.250 --> 00:22:50.400 friendliness, absolutely. 00:22:50.400 --> 00:22:55.000 Nothing is more frustrating than sitting there waiting, right? 00:22:55.000 --> 00:22:56.500 So, that's a good reason. 00:22:56.500 --> 00:22:58.666 What are some other reasons why? 00:22:58.666 --> 00:23:02.625 Sometimes they have real time constraints 00:23:02.625 --> 00:23:09.000 so they don't actually work unless they perform adequately. 00:23:09.000 --> 00:23:10.000 Yeah? 00:23:10.000 --> 00:23:13.500 Hard to get, well, we don't usually 00:23:13.500 --> 00:23:17.000 quantify user friendliness so I'm not sure, 00:23:17.000 --> 00:23:20.142 but I understand what you're saying. 00:23:20.142 --> 00:23:25.250 He said we don't get exponential performance improvements 00:23:25.250 --> 00:23:27.500 in user friendliness. 00:23:27.500 --> 00:23:34.000 We often don't get that in performance either, by the way. 00:23:34.000 --> 00:23:38.000 [LAUGHTER] Sometimes we do, but that's good. 00:23:38.000 --> 00:23:42.000 There are several reasons that I think are important. 00:23:42.000 --> 00:23:45.000 Once is that often performance measures 00:23:45.000 --> 00:23:48.600 the line between the feasible and the infeasible. 00:23:48.600 --> 00:23:51.666 We have heard some of these things. 00:23:51.666 --> 00:23:56.000 For example, when there are real time requirements, 00:23:56.000 --> 00:24:02.000 if it's not fast enough it's simply not functional. 00:24:02.000 --> 00:24:04.300 Or, if it uses too much memory it's simply not 00:24:04.300 --> 00:24:05.666 going to work for you. 00:24:05.666 --> 00:24:08.360 And, as a consequence, what you find is algorithms 00:24:08.360 --> 00:24:10.500 are on the cutting edge of entrepreneurship. 00:24:10.500 --> 00:24:13.999 If you're talking about just re implementing stuff 00:24:13.999 --> 00:24:16.428 that people did ten years ago, performance 00:24:16.428 --> 00:24:19.000 isn't that important at some level. 00:24:19.000 --> 00:24:21.541 But, if you're talking about doing stuff 00:24:21.541 --> 00:24:24.665 that nobody has done before, one of the reasons 00:24:24.665 --> 00:24:28.200 often that they haven't done it is because it's too 00:24:28.200 --> 00:24:29.000 time consuming. 00:24:29.000 --> 00:24:31.600 Things don't scale and so forth. 00:24:31.600 --> 00:24:36.000 So, that's one reason, is the feasible versus infeasible. 00:24:36.000 --> 00:24:39.108 Another thing is that algorithms give you 00:24:39.108 --> 00:24:42.000 a language for talking about program behavior, 00:24:42.000 --> 00:24:45.776 and that turns out to be a language that 00:24:45.776 --> 00:24:48.454 has been pervasive through computer science, 00:24:48.454 --> 00:24:53.400 is that the theoretical language is what gets adopted by all 00:24:53.400 --> 00:24:57.000 the practitioners because it's a clean way of thinking 00:24:57.000 --> 00:24:58.250 about things. 00:24:58.250 --> 00:25:02.454 A good way I think about performance, 00:25:02.454 --> 00:25:07.000 and the reason it's on the bottom of the heap, 00:25:07.000 --> 00:25:13.000 is sort of like performance is like money, it's like currency. 00:25:13.000 --> 00:25:17.224 You say what good does a stack of hundred dollar bills 00:25:17.224 --> 00:25:18.428 do for you? 00:25:18.428 --> 00:25:23.725 Would you rather have food or water or shelter or whatever? 00:25:23.725 --> 00:25:28.000 And you're willing to pay those hundred dollar 00:25:28.000 --> 00:25:31.600 bills, if you have hundred dollar bills, 00:25:31.600 --> 00:25:33.400 for that commodity. 00:25:33.400 --> 00:25:39.000 Even though water is far more important to your living. 00:25:39.000 --> 00:25:42.000 Well, similarly, performance is what you use 00:25:42.000 --> 00:25:44.000 to pay for user friendliness. 00:25:44.000 --> 00:25:46.178 It's what you pay for security. 00:25:46.178 --> 00:25:48.666 And you hear people say, for example, 00:25:48.666 --> 00:25:50.428 that I want greater functionality, 00:25:50.428 --> 00:25:54.362 so people will program in Java, even though it's 00:25:54.362 --> 00:25:57.086 much slower than C, because they'll 00:25:57.086 --> 00:26:00.331 say it costs me maybe a factor of three 00:26:00.331 --> 00:26:03.332 or something in performance to program in Java. 00:26:03.332 --> 00:26:06.444 But Java is worth it because it's 00:26:06.444 --> 00:26:10.333 got all these object oriented features and so forth, 00:26:10.333 --> 00:26:12.000 exception mechanisms and so on. 00:26:12.000 --> 00:26:15.663 And so people are willing to pay a factor of three 00:26:15.663 --> 00:26:16.333 in performance. 00:26:16.333 --> 00:26:18.333 So, that's why you want performance 00:26:18.333 --> 00:26:22.000 because you can use it to pay for these other things 00:26:22.000 --> 00:26:22.999 that you want. 00:26:22.999 --> 00:26:27.000 And that's why, in some sense, it's on the bottom of the heap, 00:26:27.000 --> 00:26:32.000 because it's the universal thing that you quantify. 00:26:32.000 --> 00:26:35.663 Do you want to spend a factor of two on this 00:26:35.663 --> 00:26:39.000 or spend a factor of three on security, et cetera? 00:26:39.000 --> 00:26:42.000 And, in addition, the lessons generalize 00:26:42.000 --> 00:26:46.000 to other resource measures like communication, 00:26:46.000 --> 00:26:47.815 like memory and so forth. 00:26:47.815 --> 00:26:51.142 And the last reason we study algorithm performance 00:26:51.142 --> 00:26:54.000 is it's tons of fun. 00:26:54.000 --> 00:26:56.000 Speed is always fun, right? 00:26:56.000 --> 00:27:00.000 Why do people drive fast cars, race horses, whatever? 00:27:00.000 --> 00:27:04.665 Rockets, et cetera, why do we do that? 00:27:04.665 --> 00:27:05.700 Because speed is fun. 00:27:05.700 --> 00:27:06.200 Ski. 00:27:06.200 --> 00:27:07.000 Who likes to ski? 00:27:07.000 --> 00:27:08.200 I love to ski. 00:27:08.200 --> 00:27:10.333 I like going fast on those skis. 00:27:10.333 --> 00:27:11.000 It's fun. 00:27:11.000 --> 00:27:13.000 Hockey, fast sports, right? 00:27:13.000 --> 00:27:14.800 We all like the fast sports. 00:27:14.800 --> 00:27:16.570 Not all of us, I mean. 00:27:16.570 --> 00:27:18.855 Some people say he's not talking to me. 00:27:18.855 --> 00:27:20.000 OK, let's move on. 00:27:20.000 --> 00:27:24.375 That's sort of a little bit of a notion as to why we study this, 00:27:24.375 --> 00:27:27.272 is that it does, in some sense, form 00:27:27.272 --> 00:27:30.714 a common basis for all these other things we care about. 00:27:30.714 --> 00:27:37.775 And so we want to understand how can we generate money 00:27:37.775 --> 00:27:40.000 for ourselves in computation? 00:27:40.000 --> 00:27:44.000 We're going to start out with a very simple problem. 00:27:44.000 --> 00:27:47.178 It's one of the oldest problems that 00:27:47.178 --> 00:27:52.000 has been studied in algorithms, is the problem of sorting. 00:27:52.000 --> 00:27:57.000 We're going to actually study this for several lectures 00:27:57.000 --> 00:28:03.000 because sorting contains many algorithmic techniques. 00:28:03.000 --> 00:28:10.000 The sorting problem is the following. 00:28:10.000 --> 00:28:21.142 We have a sequence a 1, a 2 up to a n of numbers as input. 00:28:21.142 --> 00:28:33.000 And our output is a permutation of those numbers. 00:28:42.000 --> 00:28:47.000 A permutation is a rearrangement of the numbers. 00:28:47.000 --> 00:28:53.000 Every number appears exactly once in the rearrangement such 00:28:53.000 --> 00:28:57.708 that, I sometimes use a dollar sign to mean "such that," 00:28:57.708 --> 00:29:03.664 a 1 is less than or equal to a 2 prime. 00:29:03.664 --> 00:29:11.000 Such that they are monotonically increasing in size. 00:29:11.000 --> 00:29:17.000 Take a bunch of numbers, put them in order. 00:29:17.000 --> 00:29:23.000 Here's an algorithm to do it. 00:29:23.000 --> 00:29:27.000 It's called insertion sort. 00:29:40.000 --> 00:29:44.500 And we will write this algorithm in what we call pseudocode. 00:29:44.500 --> 00:29:47.500 It's sort of a programming language, 00:29:47.500 --> 00:29:51.000 except it's got English in there often. 00:29:51.000 --> 00:29:57.000 And it's just a shorthand for writing for being precise. 00:29:57.000 --> 00:30:01.800 So this sorts A from 1 to n. 00:30:01.800 --> 00:30:06.000 And here is the code for it. 00:30:59.000 --> 00:31:01.000 This is what we call pseudocode. 00:31:01.000 --> 00:31:04.632 And if you don't understand the pseudocode then 00:31:04.632 --> 00:31:09.000 you should ask questions about any of the notations. 00:31:09.000 --> 00:31:13.000 You will start to get used to it as we go on. 00:31:13.000 --> 00:31:16.500 One thing is that in the pseudocode 00:31:16.500 --> 00:31:19.816 we use indentation, where in most languages 00:31:19.816 --> 00:31:24.600 they have some kind of begin end delimiters like curly braces 00:31:24.600 --> 00:31:28.332 or something in Java or C, for example. 00:31:28.332 --> 00:31:31.000 We just use indentation. 00:31:31.000 --> 00:31:33.331 The whole idea of the pseudocode is 00:31:33.331 --> 00:31:37.220 to try to get the algorithms as short as possible 00:31:37.220 --> 00:31:41.000 while still understanding what the individual steps are. 00:31:41.000 --> 00:31:44.000 In practice, there actually have been 00:31:44.000 --> 00:31:47.665 languages that use indentation as a means of showing 00:31:47.665 --> 00:31:49.000 the nesting of things. 00:31:49.000 --> 00:31:52.000 It's generally a bad idea, because if things go over one 00:31:52.000 --> 00:31:56.775 page to another, for example, you cannot tell what level 00:31:56.775 --> 00:31:59.000 of nesting it is. 00:31:59.000 --> 00:32:03.000 Whereas, with explicit braces it's much easier to tell. 00:32:03.000 --> 00:32:09.000 So, there are reasons why this is a bad notation if you were 00:32:09.000 --> 00:32:10.500 doing software engineering. 00:32:10.500 --> 00:32:15.362 But it's a good one for us because it just 00:32:15.362 --> 00:32:20.142 keeps things short and makes fewer things to write down. 00:32:20.142 --> 00:32:23.000 So, this is insertion sort. 00:32:23.000 --> 00:32:29.000 Let's try to figure out a little bit what this does. 00:32:29.000 --> 00:32:34.380 It basically takes an array A and at any point 00:32:34.380 --> 00:32:42.000 the thing to understand is, we're setting basically, 00:32:42.000 --> 00:32:48.000 we're running the outer loop from j is 2 to n, 00:32:48.000 --> 00:32:53.710 and the inner loop that starts at j minus 1 00:32:53.710 --> 00:32:57.998 and then goes down until it's zero. 00:32:57.998 --> 00:33:03.776 Basically, if we look at any point in the algorithm, 00:33:03.776 --> 00:33:07.600 we essentially are looking at some element here j. 00:33:07.600 --> 00:33:10.000 A of j, the jth element. 00:33:10.000 --> 00:33:14.708 And what we do essentially is we pull a value out 00:33:14.708 --> 00:33:16.999 here that we call the key. 00:33:16.999 --> 00:33:20.665 And at this point the important thing to understand, 00:33:20.665 --> 00:33:25.600 and we'll talk more about this in recitation on Friday, 00:33:25.600 --> 00:33:30.800 is that there is an invariant that is being maintained 00:33:30.800 --> 00:33:35.000 by this loop each time through. 00:33:35.000 --> 00:33:40.000 And the invariant is that this part of the array is sorted. 00:33:40.000 --> 00:33:45.000 And the goal each time through the loop is to increase, 00:33:45.000 --> 00:33:51.000 is to add one to the length of the things that are sorted. 00:33:51.000 --> 00:33:54.996 And the way we do that is we pull out the key 00:33:54.996 --> 00:33:58.725 and we just copy values up like this. 00:33:58.725 --> 00:34:02.384 And keep copying up until we find 00:34:02.384 --> 00:34:05.840 the place where this key goes, and then we 00:34:05.840 --> 00:34:07.856 insert it in that place. 00:34:07.856 --> 00:34:11.000 And that's why it's called insertion sort. 00:34:11.000 --> 00:34:16.270 We just sort of move the things, copy the things up 00:34:16.270 --> 00:34:21.331 until we find where it goes, and then we put it into place. 00:34:21.331 --> 00:34:25.535 And now we have it from A from one to j is sorted, 00:34:25.535 --> 00:34:28.714 and now we can work on j plus one. 00:34:28.714 --> 00:34:33.000 Let's give an example of that. 00:34:33.000 --> 00:34:38.000 Imagine we are doing 8, 2, 4, 9, 3, 6. 00:34:38.000 --> 00:34:41.500 We start out with j equals 2. 00:34:41.500 --> 00:34:47.000 And we figure out that we want to insert it there. 00:34:47.000 --> 00:34:51.080 Now we have 2, 8, 4, 9, 3, 6. 00:34:51.080 --> 00:35:00.000 Then we look at the four and say oh, well, that goes over here. 00:35:00.000 --> 00:35:03.500 We get 2, 4, 8, 9, 3, 6 after the second iteration 00:35:03.500 --> 00:35:05.500 of the outer loop. 00:35:05.500 --> 00:35:10.500 Then we look at 9 and discover immediately it just 00:35:10.500 --> 00:35:12.000 goes right there. 00:35:12.000 --> 00:35:15.000 Very little work to do on that step. 00:35:15.000 --> 00:35:20.000 So, we have exactly the same output after that iteration. 00:35:20.000 --> 00:35:24.149 Then we look at the 3 and that's going 00:35:24.149 --> 00:35:26.666 to be inserted over there. 00:35:26.666 --> 00:35:36.500 2, 3, 4, 8, 9, and that goes in there. 00:35:36.500 --> 00:35:44.000 2, 3, 4, 6, 8, 00:35:44.000 --> 00:35:47.000 Question? 00:35:58.000 --> 00:36:01.000 The array initially starts at one, yes. 00:36:01.000 --> 00:36:02.000 A[1...n], OK? 00:36:02.000 --> 00:36:05.666 So, this is the insertion sort algorithm. 00:36:05.666 --> 00:36:11.725 And it's the first algorithm that we're going to analyze. 00:36:11.725 --> 00:36:15.832 And we're going to pull out some tools 00:36:15.832 --> 00:36:18.744 that we have from our math background 00:36:18.744 --> 00:36:21.200 to help to analyze it. 00:36:21.200 --> 00:36:29.000 First of all, let's take a look at the issue of running time. 00:36:29.000 --> 00:36:33.900 The running time depends, of this algorithm 00:36:33.900 --> 00:36:37.500 depends on a lot of things. 00:36:37.500 --> 00:36:42.500 One thing it depends on is the input itself. 00:36:42.500 --> 00:36:55.500 For example, if the input is already sorted -- -- 00:36:55.500 --> 00:37:00.000 then insertion sort has very little work to do. 00:37:00.000 --> 00:37:04.000 Because every time through it's going to be like this case. 00:37:04.000 --> 00:37:07.267 It doesn't have to shuffle too many guys over 00:37:07.267 --> 00:37:09.284 because they're already in place. 00:37:09.284 --> 00:37:12.712 Whereas, in some sense, what's the worst case 00:37:12.712 --> 00:37:14.000 for insertion sort? 00:37:14.000 --> 00:37:16.499 If it is reverse sorted then it's 00:37:16.499 --> 00:37:19.726 going to have to do a lot of work 00:37:19.726 --> 00:37:23.000 because it's going to have to shuffle everything over 00:37:23.000 --> 00:37:26.227 on each step of the outer loop. 00:37:26.227 --> 00:37:30.500 In addition to the actual input it depends, of course, 00:37:30.500 --> 00:37:32.000 on the input size. 00:37:38.000 --> 00:37:41.000 Here, for example, we did six elements. 00:37:41.000 --> 00:37:47.500 It's going to take longer if we, for example, do six times 00:37:47.500 --> 00:37:50.000 ten to the ninth elements. 00:37:50.000 --> 00:37:53.424 If we were sorting a lot more stuff, 00:37:53.424 --> 00:37:57.142 it's going to take us a lot longer. 00:37:57.142 --> 00:38:01.635 Typically, the way we handle that is we 00:38:01.635 --> 00:38:06.500 are going to parameterize things in the input size. 00:38:06.500 --> 00:38:11.284 We are going to talk about time as a function 00:38:11.284 --> 00:38:14.708 of the size of things that we are 00:38:14.708 --> 00:38:19.000 sorting so we can look at what is the behavior of that. 00:38:19.000 --> 00:38:23.576 And the last thing I want to say about running time 00:38:23.576 --> 00:38:30.000 is generally we want upper bonds on the running time. 00:38:30.000 --> 00:38:34.428 We want to know that the time is no more than a certain amount. 00:38:34.428 --> 00:38:38.500 And the reason is because that represents a guarantee 00:38:38.500 --> 00:38:40.000 to the user. 00:38:40.000 --> 00:38:44.071 If I say it's not going to run, for example, if I tell 00:38:44.071 --> 00:38:46.927 you here's a program and it won't run 00:38:46.927 --> 00:38:50.000 more than three seconds, that gives you 00:38:50.000 --> 00:38:55.220 real information about how you could use it, for example, 00:38:55.220 --> 00:38:58.000 in a real time setting. 00:38:58.000 --> 00:39:00.331 Whereas, if I said here's a program 00:39:00.331 --> 00:39:02.888 and it goes at least three seconds, 00:39:02.888 --> 00:39:07.600 you don't know if it's going to go for three years. 00:39:07.600 --> 00:39:10.375 It doesn't give you that much guarantee 00:39:10.375 --> 00:39:13.000 if you are a user of it. 00:39:13.000 --> 00:39:16.552 Generally we want upper bonds because it represents 00:39:16.552 --> 00:39:20.000 a guarantee to the user. 00:39:30.000 --> 00:39:33.000 There are different kinds of analyses that people do. 00:39:44.000 --> 00:39:50.544 The one we're mostly going to focus on 00:39:50.544 --> 00:39:55.400 is what's called worst case analysis. 00:39:55.400 --> 00:40:04.996 And this is what we do usually where we define T of n 00:40:04.996 --> 00:40:12.142 to be the maximum time on any input of size n. 00:40:12.142 --> 00:40:16.535 So, it's the maximum input, the maximum it could possibly 00:40:16.535 --> 00:40:19.000 cost us on an input of size n. 00:40:19.000 --> 00:40:21.720 What that does is, if you look at the fact 00:40:21.720 --> 00:40:24.220 that sometimes the inputs are better 00:40:24.220 --> 00:40:26.333 and sometimes they're worse, we're 00:40:26.333 --> 00:40:28.664 looking at the worst case of those 00:40:28.664 --> 00:40:30.888 because that's the way we're going 00:40:30.888 --> 00:40:34.000 to be able to make a guarantee. 00:40:34.000 --> 00:40:36.664 It always does something rather than just sometimes 00:40:36.664 --> 00:40:37.500 does something. 00:40:37.500 --> 00:40:40.285 So, we're looking at the maximum. 00:40:40.285 --> 00:40:44.664 Notice that if I didn't have maximum then T(n) in some sense 00:40:44.664 --> 00:40:47.800 is a relation, not a function, because the time 00:40:47.800 --> 00:40:52.000 on an input of size n depends on which input of size n. 00:40:52.000 --> 00:40:54.400 I could have many different times, 00:40:54.400 --> 00:40:56.999 but by putting the maximum at it, 00:40:56.999 --> 00:40:59.400 it turns that relation into a function 00:40:59.400 --> 00:41:04.333 because there's only one maximum time that it will take. 00:41:04.333 --> 00:41:13.856 Sometimes we will talk about average case. 00:41:13.856 --> 00:41:21.000 Sometimes we will do this. 00:41:21.000 --> 00:41:36.332 Here T of n is then the expected time over all inputs of size n. 00:41:36.332 --> 00:41:39.000 It's the expected time. 00:41:39.000 --> 00:41:43.280 Now, if I talk about expected time, what else do 00:41:43.280 --> 00:41:45.400 I need to say here? 00:41:45.400 --> 00:41:48.000 What does that mean, expected time? 00:41:48.000 --> 00:41:49.000 I'm sorry. 00:41:49.000 --> 00:41:50.800 Raise your hand. 00:41:50.800 --> 00:41:52.000 Expected inputs. 00:41:52.000 --> 00:41:56.000 What does that mean, expected inputs? 00:42:05.000 --> 00:42:06.816 I need more math. 00:42:06.816 --> 00:42:10.888 What do I need by expected time here, math? 00:42:10.888 --> 00:42:15.142 You have to take the time of every input 00:42:15.142 --> 00:42:18.000 and then average them, OK. 00:42:18.000 --> 00:42:22.000 That's kind of what we mean by expected time. 00:42:22.000 --> 00:42:22.666 Good. 00:42:22.666 --> 00:42:24.000 Not quite. 00:42:24.000 --> 00:42:28.000 I mean, what you say is completely correct, 00:42:28.000 --> 00:42:32.165 except is not quite enough. 00:42:32.165 --> 00:42:33.000 Yeah? 00:42:33.000 --> 00:42:37.905 It's the time of every input times the probability 00:42:37.905 --> 00:42:40.816 that it will be that input. 00:42:40.816 --> 00:42:45.665 It's a way of taking a weighted average, exactly right. 00:42:45.665 --> 00:42:51.270 How do I know what the probability of every input is? 00:42:51.270 --> 00:42:57.200 How do I know what the probability a particular input 00:42:57.200 --> 00:43:02.000 occurs is in a given situation? 00:43:02.000 --> 00:43:03.000 I don't. 00:43:03.000 --> 00:43:06.000 I have to make an assumption. 00:43:06.000 --> 00:43:08.400 What's that assumption called? 00:43:08.400 --> 00:43:13.875 What kind of assumption do I have to meet? 00:43:13.875 --> 00:43:26.855 I need an assumption -- -- of the statistical distribution 00:43:26.855 --> 00:43:28.000 of inputs. 00:43:28.000 --> 00:43:31.750 Otherwise, expected time doesn't mean anything 00:43:31.750 --> 00:43:38.000 because I don't know what the probability of something is. 00:43:38.000 --> 00:43:42.500 In order to do probability, you need some assumptions 00:43:42.500 --> 00:43:48.000 and you've got to state those assumptions clearly. 00:43:48.000 --> 00:43:51.000 One of the most common assumptions 00:43:51.000 --> 00:43:54.713 is that all inputs are equally likely. 00:43:54.713 --> 00:43:57.571 That's called the uniform distribution. 00:43:57.571 --> 00:44:04.000 Every input of size n is equally likely, that kind of thing. 00:44:04.000 --> 00:44:09.000 But there are other ways that you could make that assumption, 00:44:09.000 --> 00:44:11.912 and they may not all be true. 00:44:11.912 --> 00:44:15.600 This is much more complicated, as you can see. 00:44:15.600 --> 00:44:20.000 Fortunately, all of you have a strong probability background. 00:44:20.000 --> 00:44:23.600 And so we will not have any trouble addressing 00:44:23.600 --> 00:44:27.000 these probabilistic issues of dealing 00:44:27.000 --> 00:44:30.000 with expectations and such. 00:44:30.000 --> 00:44:34.000 If you don't, time to go and say gee, 00:44:34.000 --> 00:44:38.200 maybe I should take that Probability class 00:44:38.200 --> 00:44:42.180 that is a prerequisite for this class. 00:44:42.180 --> 00:44:49.180 The last one I am going to mention is best case analysis. 00:44:49.180 --> 00:44:53.000 And this I claim is bogus. 00:44:53.000 --> 00:44:53.666 Bogus. 00:44:53.666 --> 00:44:55.000 No good. 00:44:55.000 --> 00:44:59.834 Why is best-case analysis bogus? 00:44:59.834 --> 00:45:00.333 Yeah? 00:45:00.333 --> 00:45:03.200 The best case probably doesn't ever happen. 00:45:03.200 --> 00:45:06.816 Actually, it's interesting because for the sorting 00:45:06.816 --> 00:45:10.625 problem, the most common things that get sorted 00:45:10.625 --> 00:45:15.000 are things that are already sorted interestingly, 00:45:15.000 --> 00:45:17.140 or at least almost sorted. 00:45:17.140 --> 00:45:21.744 For example, one of the most common things that are sorted 00:45:21.744 --> 00:45:23.570 is check numbers by banks. 00:45:23.570 --> 00:45:28.750 They tend to come in, in the same order that they 00:45:28.750 --> 00:45:30.000 are written. 00:45:30.000 --> 00:45:36.000 They're sorting things that are almost always sorted. 00:45:36.000 --> 00:45:38.856 I mean, it's good. 00:45:38.856 --> 00:45:42.665 When upper bond, not lower bound? 00:45:42.665 --> 00:45:46.500 Yeah, you want to make a guarantee. 00:45:46.500 --> 00:45:51.000 And so why is this not a guarantee? 00:45:51.000 --> 00:46:01.125 You're onto something there, but we need a little more precision 00:46:01.125 --> 00:46:02.000 here. 00:46:02.000 --> 00:46:03.501 How can I cheat? 00:46:03.501 --> 00:46:04.000 Yeah? 00:46:04.000 --> 00:46:06.000 Yeah, you can cheat. 00:46:06.000 --> 00:46:07.000 You cheat. 00:46:07.000 --> 00:46:11.664 You take any slow algorithm that you want 00:46:11.664 --> 00:46:15.875 and just check for some particular input, 00:46:15.875 --> 00:46:23.200 and if it's that input, then you say immediately yeah, OK, here 00:46:23.200 --> 00:46:25.000 is the answer. 00:46:25.000 --> 00:46:30.000 And then it's got a good best case. 00:46:30.000 --> 00:46:35.830 But I didn't tell you anything about the vast majority 00:46:35.830 --> 00:46:38.500 of what is going on. 00:46:38.500 --> 00:46:42.571 So, you can cheat with a slow algorithm 00:46:42.571 --> 00:46:46.000 that works fast on some input. 00:46:46.000 --> 00:46:50.081 It doesn't really do much for you 00:46:50.081 --> 00:46:54.500 so we normally don't worry about that. 00:46:54.500 --> 00:46:56.000 Let's see. 00:46:56.000 --> 00:47:02.000 What is insertion sorts worst case time? 00:47:02.000 --> 00:47:07.000 Now we get into some sort of funny issues. 00:47:07.000 --> 00:47:12.833 First of all, it sort of depends on the computer 00:47:12.833 --> 00:47:15.332 you're running on. 00:47:15.332 --> 00:47:17.625 Whose computer, right? 00:47:17.625 --> 00:47:24.499 Is it a big supercomputer or is it your wristwatch? 00:47:24.499 --> 00:47:29.000 They have different computational abilities. 00:47:29.000 --> 00:47:34.428 And when we compare algorithms, we 00:47:34.428 --> 00:47:37.000 compare them typically for relative speed. 00:47:37.000 --> 00:47:42.000 This is if you compared two algorithms on the same machine. 00:47:42.000 --> 00:47:44.625 You could argue, well, it doesn't really 00:47:44.625 --> 00:47:48.328 matter what the machine is because I will just 00:47:48.328 --> 00:47:50.500 look at their relative speed. 00:47:50.500 --> 00:47:55.000 But, of course, I may also be interested in absolute speed. 00:47:55.000 --> 00:47:58.108 Is one algorithm actually better no matter 00:47:58.108 --> 00:48:02.000 what machine it's run on? 00:48:08.000 --> 00:48:11.213 And so this kind of gets sort of confusing 00:48:11.213 --> 00:48:14.920 as to how I can talk about the worst case 00:48:14.920 --> 00:48:18.500 time of an algorithm of a piece of software 00:48:18.500 --> 00:48:23.000 when I am not talking about the hardware because, 00:48:23.000 --> 00:48:27.000 clearly, if I had run on a faster machine, 00:48:27.000 --> 00:48:30.000 my algorithms are going to go faster. 00:48:30.000 --> 00:48:36.000 So, this is where you get the big idea of algorithms. 00:48:36.000 --> 00:48:39.000 Which is why algorithm is such a huge field, 00:48:39.000 --> 00:48:43.000 why it spawns companies like Google, like Akamai, 00:48:43.000 --> 00:48:44.200 like Amazon. 00:48:44.200 --> 00:48:47.200 Why algorithmic analysis, throughout the history 00:48:47.200 --> 00:48:50.416 of computing, has been such a huge success, 00:48:50.416 --> 00:48:53.744 is our ability to master and to be 00:48:53.744 --> 00:48:57.775 able to take what is apparently a really 00:48:57.775 --> 00:49:02.220 messy, complicated situation and reduce it to being 00:49:02.220 --> 00:49:05.000 able to do some mathematics. 00:49:05.000 --> 00:49:09.000 And that idea is called asymptotic analysis. 00:49:17.000 --> 00:49:21.000 And the basic idea of asymptotic analysis is to ignore 00:49:21.000 --> 00:49:34.500 machine-dependent constants -- -- 00:49:34.500 --> 00:49:38.000 and, instead of the actual running time, 00:49:38.000 --> 00:49:43.000 look at the growth of the running time. 00:49:59.000 --> 00:50:02.000 So, we don't look at the actual running time. 00:50:02.000 --> 00:50:04.080 We look at the growth. 00:50:04.080 --> 00:50:07.000 Let's see what we mean by that. 00:50:07.000 --> 00:50:08.500 This is a huge idea. 00:50:08.500 --> 00:50:11.000 It's not a hard idea, otherwise I 00:50:11.000 --> 00:50:16.000 wouldn't be able to teach it in the first lecture, 00:50:16.000 --> 00:50:17.665 but it's a huge idea. 00:50:17.665 --> 00:50:21.110 We are going to spend a couple of lectures 00:50:21.110 --> 00:50:23.885 understanding the implications of that 00:50:23.885 --> 00:50:30.000 and will basically be doing it throughout the term. 00:50:30.000 --> 00:50:33.000 And if you go on to be practicing engineers, 00:50:33.000 --> 00:50:36.000 you will be doing it all the time. 00:50:36.000 --> 00:50:39.600 In order to do that, we adopt some notations 00:50:39.600 --> 00:50:42.140 that are going to help us. 00:50:42.140 --> 00:50:46.000 In particular, we will adopt asymptotic notation. 00:50:46.000 --> 00:50:51.000 Most of you have seen some kind of asymptotic notation. 00:50:51.000 --> 00:50:53.997 Maybe a few of you haven't, but mostly you 00:50:53.997 --> 00:50:56.200 should have seen a little bit. 00:50:56.200 --> 00:51:01.571 The one we're going to be using in this class 00:51:01.571 --> 00:51:05.000 predominantly is theta notation. 00:51:05.000 --> 00:51:09.900 And theta notation is pretty easy notation 00:51:09.900 --> 00:51:16.000 to master because all you do is, from a formula, 00:51:16.000 --> 00:51:24.000 just drop low order terms and ignore leading constants. 00:51:30.000 --> 00:51:37.500 For example, if I have a formula like 3n^3 = 90n^2 - 5n + 6046, 00:51:37.500 --> 00:51:43.494 I say, well, what low order terms do I drop? 00:51:43.494 --> 00:51:48.500 Well, n^3 is a bigger term n^2 than. 00:51:48.500 --> 00:51:56.000 I am going to drop all these terms and ignore the leading 00:51:56.000 --> 00:52:01.000 constant, so I say that's Theta(n^3). 00:52:01.000 --> 00:52:04.000 That's pretty easy. 00:52:04.000 --> 00:52:06.000 So, that's theta notation. 00:52:06.000 --> 00:52:11.000 Now, this is an engineering way of manipulating theta notation. 00:52:11.000 --> 00:52:14.000 There is actually a mathematical definition 00:52:14.000 --> 00:52:18.000 for this, which we are going to talk 00:52:18.000 --> 00:52:22.000 about next time, which is a definition in terms 00:52:22.000 --> 00:52:23.200 of sets of functions. 00:52:23.200 --> 00:52:25.500 And, you are going to be responsible, 00:52:25.500 --> 00:52:30.571 this is both a math and a computer science engineering 00:52:30.571 --> 00:52:31.142 class. 00:52:31.142 --> 00:52:34.416 Throughout the course you are going 00:52:34.416 --> 00:52:37.328 to be responsible both for mathematical rigor 00:52:37.328 --> 00:52:41.220 as if it were a math course and engineering 00:52:41.220 --> 00:52:43.666 commonsense because it's an engineering course. 00:52:43.666 --> 00:52:46.000 We are going to be doing both. 00:52:46.000 --> 00:52:50.000 This is the engineering way of understanding what you do, 00:52:50.000 --> 00:52:54.000 so you're responsible for being able to do these manipulations. 00:52:54.000 --> 00:52:57.000 You're also going to be responsible for understanding 00:52:57.000 --> 00:52:59.400 the mathematical definition of theta notion 00:52:59.400 --> 00:53:03.500 and of its related O notation and omega notation. 00:53:03.500 --> 00:53:09.332 If I take a look as n approached infinity, 00:53:09.332 --> 00:53:16.400 a Theta(n^2) algorithm always beats, eventually, 00:53:16.400 --> 00:53:20.000 a Theta(n^3) algorithm. 00:53:20.000 --> 00:53:27.600 As n gets bigger, it doesn't matter what these other terms 00:53:27.600 --> 00:53:34.636 were if I were describing the absolute precise behavior 00:53:34.636 --> 00:53:37.816 in terms of a formula. 00:53:37.816 --> 00:53:44.000 If I had a Theta(n^2) algorithm, it would always be faster 00:53:44.000 --> 00:53:47.776 for sufficiently large n than a Theta(n^3) algorithm. 00:53:47.776 --> 00:53:51.776 It wouldn't matter what those low order terms were. 00:53:51.776 --> 00:53:55.665 It wouldn't matter what the leading constant was. 00:53:55.665 --> 00:53:59.000 This one will always be faster. 00:53:59.000 --> 00:54:04.500 Even if you ran the Theta(n^2) algorithm on a slow computer 00:54:04.500 --> 00:54:09.000 and the Theta(n^3) algorithm on a fast computer. 00:54:09.000 --> 00:54:12.000 The great thing about asymptotic notation 00:54:12.000 --> 00:54:16.000 is it satisfies our issue of being 00:54:16.000 --> 00:54:20.500 able to compare both relative and absolute speed, 00:54:20.500 --> 00:54:25.666 because we are able to do this no matter 00:54:25.666 --> 00:54:29.000 what the computer platform. 00:54:29.000 --> 00:54:34.000 On different platforms we may get different constants here, 00:54:34.000 --> 00:54:39.000 machine dependent constants for the actual running time, 00:54:39.000 --> 00:54:44.564 but if I look at the growth as the size of the input 00:54:44.564 --> 00:54:49.000 gets larger, the asymptotics generally won't change. 00:54:49.000 --> 00:54:53.444 For example, I will just draw that as a picture. 00:54:53.444 --> 00:54:59.284 If I have n on this axis and T(n) on this axis. 00:54:59.284 --> 00:55:04.000 This may be, for example, a Theta(n^3) algorithm and this 00:55:04.000 --> 00:55:06.270 may be a Theta(n^2) algorithm. 00:55:06.270 --> 00:55:12.500 There is always going to be some point n o where for everything 00:55:12.500 --> 00:55:17.500 larger the Theta(n^2) algorithm is going to be cheaper than 00:55:17.500 --> 00:55:21.496 the Theta(n^3) algorithm not matter how much advantage you 00:55:21.496 --> 00:55:26.180 give it at the beginning in terms of the speed 00:55:26.180 --> 00:55:30.000 of the computer you are running on. 00:55:30.000 --> 00:55:32.541 Now, from an engineering point of view, 00:55:32.541 --> 00:55:36.664 there are some issues we have to deal with because sometimes it 00:55:36.664 --> 00:55:40.997 could be that that n o is so large that the computers aren't 00:55:40.997 --> 00:55:43.500 big enough to run the problem. 00:55:43.500 --> 00:55:47.140 That's why we, nevertheless, are interested in some 00:55:47.140 --> 00:55:51.000 of the slower algorithms, because some of the slower 00:55:51.000 --> 00:55:55.000 algorithms, even though they may not asymptotically be slower, 00:55:55.000 --> 00:56:00.000 I mean asymptotically they will be slower. 00:56:00.000 --> 00:56:03.000 They may still be faster on reasonable sizes of things. 00:56:03.000 --> 00:56:07.000 And so we have to both balance our mathematical understanding 00:56:07.000 --> 00:56:10.000 with our engineering commonsense in order 00:56:10.000 --> 00:56:12.000 to do good programming. 00:56:12.000 --> 00:56:14.625 So, just having done analysis of algorithms 00:56:14.625 --> 00:56:18.000 doesn't automatically make you a good programmer. 00:56:18.000 --> 00:56:21.070 You also need to learn how to program and use 00:56:21.070 --> 00:56:23.089 these tools in practice to understand 00:56:23.089 --> 00:56:27.332 when they are relevant and when they are not relevant. 00:56:27.332 --> 00:56:30.000 There is a saying. 00:56:30.000 --> 00:56:32.664 If you want to be a good program, 00:56:32.664 --> 00:56:35.776 you just program ever day for two years, 00:56:35.776 --> 00:56:38.444 you will be an excellent programmer. 00:56:38.444 --> 00:56:42.444 If you want to be a world class programmer, 00:56:42.444 --> 00:56:46.000 you can program every day for ten years, 00:56:46.000 --> 00:56:49.744 or you can program every day for two years 00:56:49.744 --> 00:56:51.888 and take an algorithms class. 00:56:51.888 --> 00:56:55.833 Let's get back to what we were doing, 00:56:55.833 --> 00:57:00.000 which is analyzing insertion sort. 00:57:00.000 --> 00:57:02.000 We are going to look at the worse case. 00:57:16.000 --> 00:57:21.000 Which, as we mentioned before, is when the input is reverse 00:57:21.000 --> 00:57:21.500 sorted. 00:57:21.500 --> 00:57:25.816 The biggest element comes first and the smallest last 00:57:25.816 --> 00:57:30.714 because now every time you do the insertion you've 00:57:30.714 --> 00:57:35.000 got to shuffle everything over. 00:57:35.000 --> 00:57:36.750 You can write down the running time 00:57:36.750 --> 00:57:38.444 by looking at the nesting of loops. 00:57:38.444 --> 00:57:40.000 What we do is we sum up. 00:57:40.000 --> 00:57:42.331 What we assume is that every operation, 00:57:42.331 --> 00:57:45.000 every elemental operation is going to take 00:57:45.000 --> 00:57:47.000 some constant amount of time. 00:57:47.000 --> 00:57:49.176 But we don't have to worry about what 00:57:49.176 --> 00:57:51.875 that constant is because we're going to be 00:57:51.875 --> 00:57:53.000 doing asymptotic analysis. 00:57:53.000 --> 00:57:54.840 As I say, the beautify of the method 00:57:54.840 --> 00:57:57.500 is that it causes all these things that 00:57:57.500 --> 00:58:01.000 are real distinctions to sort of vanish. 00:58:01.000 --> 00:58:03.997 We sort of look at them from 30,000 feet 00:58:03.997 --> 00:58:06.776 rather than from three millimeters or something. 00:58:06.776 --> 00:58:10.500 Each of these operations is going to sort of 00:58:10.500 --> 00:58:12.000 be a basic operation. 00:58:12.000 --> 00:58:16.600 One way to think about this, in terms of counting operations, 00:58:16.600 --> 00:58:19.000 is counting memory references. 00:58:19.000 --> 00:58:23.000 How many times do you actually access some variable? 00:58:23.000 --> 00:58:29.000 That's another way of sort of thinking about this model. 00:58:29.000 --> 00:58:33.800 When we do that, well, we're going to go through this loop, 00:58:33.800 --> 00:58:37.800 j is going from 2 to n, and then we're 00:58:37.800 --> 00:58:43.000 going to add up the work that we do within the loop. 00:58:43.000 --> 00:58:48.136 We can sort of write that in math as summation of j 00:58:48.136 --> 00:58:49.888 equals 2 to n. 00:58:49.888 --> 00:58:55.304 And then what is the work that is going on in this loop? 00:58:55.304 --> 00:58:59.362 Well, the work that is going on in this loop 00:58:59.362 --> 00:59:05.100 varies, but in the worst case how many operations are going 00:59:05.100 --> 00:59:10.000 on here for each value of j? 00:59:10.000 --> 00:59:17.600 For a given value of j, how much work goes on in this loop? 00:59:17.600 --> 00:59:20.000 Can somebody tell me? 00:59:20.000 --> 00:59:21.000 Asymptotically. 00:59:21.000 --> 00:59:28.544 It's j times some constant, so it's theta j. 00:59:28.544 --> 00:59:33.768 So, there is theta j work going on here 00:59:33.768 --> 00:59:38.000 because this loop starts out with i being j minus 1, 00:59:38.000 --> 00:59:43.450 and then it's doing just a constant amount of stuff 00:59:43.450 --> 00:59:47.927 for each step of the value of i, and i is running 00:59:47.927 --> 00:59:50.664 from j minus one down to zero. 00:59:50.664 --> 00:59:57.330 So, we can say that is theta j work that is going on. 00:59:57.330 --> 00:59:59.831 Do people follow that? 00:59:59.831 --> 01:00:00.330 OK. 01:00:00.330 --> 01:00:03.665 And now we have a formula we can evaluate. 01:00:03.665 --> 01:00:05.713 What is the evaluation? 01:00:05.713 --> 01:00:11.000 If I want to simplify this formula, what is that equal to? 01:00:19.901 --> 01:00:20.400 Sorry. 01:00:20.400 --> 01:00:22.000 In the back there. 01:00:28.000 --> 01:00:29.141 Yeah. 01:00:29.141 --> 01:00:30.282 OK. 01:00:30.282 --> 01:00:34.125 That's just Theta(n^2), good. 01:00:34.125 --> 01:00:38.312 Because when you're saying is the sum of consecutive numbers, 01:00:38.312 --> 01:00:39.564 you mean what? 01:00:39.564 --> 01:00:41.664 What's the mathematic term we have 01:00:41.664 --> 01:00:43.876 for that so we can communicate? 01:00:43.876 --> 01:00:48.158 You've got to know these things so you can communicate. 01:00:48.158 --> 01:00:50.569 It's called what type of sequence? 01:00:50.569 --> 01:00:53.479 It's actually a series, but that's OK. 01:00:53.479 --> 01:00:56.127 What type of series is this called? 01:00:56.127 --> 01:00:57.733 Arithmetic series, good. 01:00:57.733 --> 01:01:02.412 Wow, we've got some sharp people who can communicate. 01:01:02.412 --> 01:01:05.943 This is an arithmetic series. 01:01:05.943 --> 01:01:11.937 You're basically summing 1 + 2 + 3 + 4, some constants in there, 01:01:11.937 --> 01:01:17.210 but basically it's 1 + 2 + 3 + 4 + 5 + 6 up to n. 01:01:17.210 --> 01:01:18.376 That's Theta(n^2). 01:01:18.376 --> 01:01:25.043 If you don't know this math, there is a chapter in the book, 01:01:25.043 --> 01:01:28.974 or you could have taken the prerequisite. 01:01:28.974 --> 01:01:31.390 Arithmetic series. 01:01:31.390 --> 01:01:33.951 People have this vague recollection. 01:01:33.951 --> 01:01:34.476 Oh, yeah. 01:01:34.476 --> 01:01:34.975 Good. 01:01:34.975 --> 01:01:38.048 Now, you have to learn these manipulations. 01:01:38.048 --> 01:01:40.592 We will talk about a bit next time, 01:01:40.592 --> 01:01:43.452 but you have to learn your theta manipulations 01:01:43.452 --> 01:01:45.804 for what works with theta. 01:01:45.804 --> 01:01:48.765 And you have to be very careful because theta 01:01:48.765 --> 01:01:50.231 is a weak notation. 01:01:50.231 --> 01:01:53.881 A strong notation is something like Leibniz notation 01:01:53.881 --> 01:01:57.354 from calculus where the chain rule is just 01:01:57.354 --> 01:01:58.857 canceling two things. 01:01:58.857 --> 01:02:03.544 It's just fabulous that you can cancel in the chain rule. 01:02:03.544 --> 01:02:06.820 And Leibniz notation just expresses that so directly you 01:02:06.820 --> 01:02:07.599 can manipulate. 01:02:07.599 --> 01:02:09.468 Theta notation is not like that. 01:02:09.468 --> 01:02:12.303 If you think it is like that you are in trouble. 01:02:12.303 --> 01:02:15.033 You really have to think of what is going on 01:02:15.033 --> 01:02:16.162 under the theta notation. 01:02:16.162 --> 01:02:18.644 And it is more of a descriptive notation 01:02:18.644 --> 01:02:20.867 than it is a manipulative notation. 01:02:20.867 --> 01:02:23.363 There are manipulations you can do with it, 01:02:23.363 --> 01:02:25.661 but unless you understand what is really 01:02:25.661 --> 01:02:29.325 going on under the theta notation you will find yourself 01:02:29.325 --> 01:02:30.179 in trouble. 01:02:30.179 --> 01:02:34.575 And next time we will talk a little bit more 01:02:34.575 --> 01:02:35.977 about theta notation. 01:02:35.977 --> 01:02:38.000 Is insertion sort fast? 01:02:49.000 --> 01:02:53.000 Well, it turns out for small n it is moderately fast. 01:03:02.000 --> 01:03:11.000 But it is not at all for large n. 01:03:18.000 --> 01:03:21.626 So, I am going to give you an algorithm that is faster. 01:03:21.626 --> 01:03:22.942 It's called merge sort. 01:03:22.942 --> 01:03:26.165 I wonder if I should leave insertion sort up. 01:03:26.165 --> 01:03:27.000 Why not. 01:03:46.000 --> 01:03:49.768 I am going to write on this later, 01:03:49.768 --> 01:03:56.099 so if you are taking notes, leave some space on the left. 01:03:56.099 --> 01:04:02.000 Here is merge sort of an array A from 1 up to n. 01:04:02.000 --> 01:04:05.963 And it is basically three steps. 01:04:05.963 --> 01:04:10.079 If n equals 1 we are done. 01:04:10.079 --> 01:04:14.484 Sorting one element, it is already sorted. 01:04:14.484 --> 01:04:15.808 All right. 01:04:15.808 --> 01:04:17.310 Recursive algorithm. 01:04:17.310 --> 01:04:24.962 Otherwise, what we do is we recursively sort A from 1 01:04:24.962 --> 01:04:30.000 up to the ceiling of n over 2. 01:04:30.000 --> 01:04:39.102 And the array A of the ceiling of n over 2 plus one up to n. 01:04:39.102 --> 01:04:44.502 So, we sort two halves of the input. 01:04:44.502 --> 01:04:53.502 And then, three, we take those two lists that we have done 01:04:53.502 --> 01:04:57.000 and we merge them. 01:05:03.000 --> 01:05:05.601 And, to do that, we use a merge subroutine 01:05:05.601 --> 01:05:07.000 which I will show you. 01:05:14.000 --> 01:05:20.492 The key subroutine here is merge, and it works like this. 01:05:20.492 --> 01:05:22.388 I have two lists. 01:05:22.388 --> 01:05:25.710 Let's say one of them is 20. 01:05:25.710 --> 01:05:30.000 I am doing this in reverse order. 01:05:30.000 --> 01:05:32.682 I have sorted this like this. 01:05:32.682 --> 01:05:35.368 And then I sort another one. 01:05:35.368 --> 01:05:39.795 I don't know why I do it this order, but anyway. 01:05:39.795 --> 01:05:41.710 Here is my other list. 01:05:41.710 --> 01:05:45.445 I have my two lists that I have sorted. 01:05:45.445 --> 01:05:50.608 So, this is AA[1] to AA[|n/2|] and AA[|n/2|+1] to AA[n] for the way 01:05:50.608 --> 01:05:54.110 it will be called in this program. 01:05:54.110 --> 01:05:59.286 And now to merge these two, what I want to do 01:05:59.286 --> 01:06:04.000 is produce a sorted list out of both of them. 01:06:04.000 --> 01:06:07.900 What I do is first observe where is the smallest 01:06:07.900 --> 01:06:11.679 element of any two lists that are already sorted? 01:06:11.679 --> 01:06:15.783 It's in one of two places, the head of the first list 01:06:15.783 --> 01:06:18.387 or the head of the second list. 01:06:18.387 --> 01:06:23.028 I look at those two elements and say which one is smaller? 01:06:23.028 --> 01:06:24.613 This one is smaller. 01:06:24.613 --> 01:06:28.273 Then what I do is output into my output array 01:06:28.273 --> 01:06:30.263 the smaller of the two. 01:06:30.263 --> 01:06:32.464 And I cross it off. 01:06:32.464 --> 01:06:35.702 And now where is the next smallest element? 01:06:35.702 --> 01:06:40.320 And the answer is it's going to be the head of one of these two 01:06:40.320 --> 01:06:40.819 lists. 01:06:40.819 --> 01:06:44.237 Then I cross out this guy and put him here 01:06:44.237 --> 01:06:45.648 and circle this one. 01:06:45.648 --> 01:06:47.958 Now I look at these two guys. 01:06:47.958 --> 01:06:52.004 This one is smaller so I output that and circle that one. 01:06:52.004 --> 01:06:55.131 Now I look at these two guys, output 9. 01:06:55.131 --> 01:06:58.893 So, every step here is some fixed number of operations 01:06:58.893 --> 01:07:04.729 that is independent of the size of the arrays at each step. 01:07:04.729 --> 01:07:10.474 Each individual step is just me looking at two elements 01:07:10.474 --> 01:07:15.483 and picking out the smallest and advancing some pointers 01:07:15.483 --> 01:07:19.747 into the array so that I know where 01:07:19.747 --> 01:07:23.522 the current head of that list is. 01:07:23.522 --> 01:07:30.000 And so, therefore, the time is order n on n total elements. 01:07:30.000 --> 01:07:34.235 The time to actually go through this and merge two lists 01:07:34.235 --> 01:07:35.470 is order n. 01:07:35.470 --> 01:07:40.031 We sometimes call this linear time because it's not quadratic 01:07:40.031 --> 01:07:41.012 or whatever. 01:07:41.012 --> 01:07:45.401 It is proportional to n, proportional to the input size. 01:07:45.401 --> 01:07:46.502 It's linear time. 01:07:46.502 --> 01:07:50.098 I go through and just do this simple operation, 01:07:50.098 --> 01:07:54.291 just working up these lists, and in the end 01:07:54.291 --> 01:07:56.733 I have done essentially n operations, 01:07:56.733 --> 01:08:02.000 order n operations each of which cost constant time. 01:08:02.000 --> 01:08:05.451 That's a total of order n time. 01:08:05.451 --> 01:08:06.871 Everybody with me? 01:08:06.871 --> 01:08:07.371 OK. 01:08:07.371 --> 01:08:10.099 So, this is a recursive program. 01:08:10.099 --> 01:08:14.503 We can actually now write what is called 01:08:14.503 --> 01:08:17.309 a recurrence for this program. 01:08:17.309 --> 01:08:21.885 The way we do that is say let's let the time 01:08:21.885 --> 01:08:24.759 to sort n elements to be T(n). 01:08:24.759 --> 01:08:30.000 Then how long does it take to do step one? 01:08:35.000 --> 01:08:36.300 That's just constant time. 01:08:36.300 --> 01:08:41.370 We just check to see if n is 1, and if it is we return. 01:08:41.370 --> 01:08:45.380 That's independent of the size of anything that we are doing. 01:08:45.380 --> 01:08:48.856 It just takes a certain number of machine instructions 01:08:48.856 --> 01:08:52.875 on whatever machine and we say it is constant time. 01:08:52.875 --> 01:08:54.732 We call that theta one. 01:08:54.732 --> 01:09:00.000 This is actually a little bit of an abuse if you get into it. 01:09:00.000 --> 01:09:04.092 And the reason is because typically in order to say it 01:09:04.092 --> 01:09:07.206 you need to say what it is growing with. 01:09:07.206 --> 01:09:10.945 Nevertheless, we use this as an abuse of the notation 01:09:10.945 --> 01:09:13.550 just to mean it is a constant. 01:09:13.550 --> 01:09:16.604 So, that's an abuse just so people know. 01:09:16.604 --> 01:09:20.835 But it simplifies things if I can just write theta one. 01:09:20.835 --> 01:09:23.733 And it basically means the same thing. 01:09:23.733 --> 01:09:26.866 Now we recursively sort these two things. 01:09:26.866 --> 01:09:29.701 How can I describe that? 01:09:29.701 --> 01:09:34.674 The time to do this, I can describe recursively 01:09:34.674 --> 01:09:40.004 as T of ceiling of n over 2 plus T 01:09:40.004 --> 01:09:44.582 of n minus ceiling of n over 2. 01:09:44.582 --> 01:09:52.626 That is actually kind of messy, so what we will do is just be 01:09:52.626 --> 01:09:54.915 sloppy and write 2T(n/2). 01:09:54.915 --> 01:10:00.000 So, this is just us being sloppy. 01:10:00.000 --> 01:10:02.352 And we will see on Friday in recitation 01:10:02.352 --> 01:10:04.534 that it is OK to be sloppy. 01:10:04.534 --> 01:10:06.979 That's the great thing about algorithms. 01:10:06.979 --> 01:10:09.913 As long as you are rigorous and precise, 01:10:09.913 --> 01:10:12.502 you can be as sloppy as you want. 01:10:12.502 --> 01:10:15.846 [LAUGHTER] This is sloppy because I didn't worry 01:10:15.846 --> 01:10:19.051 about what was going on, because it turns out 01:10:19.051 --> 01:10:20.882 it doesn't make any difference. 01:10:20.882 --> 01:10:25.798 And we are going to actually see that that is the case. 01:10:25.798 --> 01:10:28.660 And, finally, I have to merge the two sorted lists which 01:10:28.660 --> 01:10:30.058 have a total of n elements. 01:10:30.058 --> 01:10:32.576 And we just analyze that using the merge subroutine. 01:10:32.576 --> 01:10:35.000 And that takes us to theta n time. 01:10:40.000 --> 01:10:43.933 That allows us now to write a recurrence for the performance 01:10:43.933 --> 01:10:45.000 of merge sort. 01:10:57.000 --> 01:11:03.032 Which is to say that T of n is equal to theta 1 01:11:03.032 --> 01:11:09.488 if n equals 1 and 2T of n over 2 plus theta of n 01:11:09.488 --> 01:11:12.250 if n is bigger than 1. 01:11:12.250 --> 01:11:16.324 Because either I am doing step one 01:11:16.324 --> 01:11:21.977 or I am doing all steps one, two and three. 01:11:21.977 --> 01:11:28.742 Here I am doing step one and I return and I am done. 01:11:28.742 --> 01:11:33.397 Or else I am doing step one, I don't return, 01:11:33.397 --> 01:11:36.730 and then I also do steps two and three. 01:11:36.730 --> 01:11:38.808 So, I add those together. 01:11:38.808 --> 01:11:43.456 I could say theta n plus theta 1, but theta n plus theta 1 01:11:43.456 --> 01:11:47.475 is just theta n because theta 1 is a lower order 01:11:47.475 --> 01:11:50.831 term than theta n and I can throw it away. 01:11:50.831 --> 01:11:56.111 It is either theta 1 or it is 2T of n over 2 plus theta n. 01:11:56.111 --> 01:11:59.051 Now, typically we won't be writing this. 01:11:59.051 --> 01:12:01.478 Usually we omit this. 01:12:01.478 --> 01:12:05.631 If it makes no difference to the solution of the recurrence, 01:12:05.631 --> 01:12:08.446 we will usually omit constant base cases. 01:12:08.446 --> 01:12:11.691 In algorithms, it's not true generally in mathematics, 01:12:11.691 --> 01:12:15.123 but in algorithms if you are running something 01:12:15.123 --> 01:12:19.145 on a constant size input it takes constant time always. 01:12:19.145 --> 01:12:22.172 So, we don't worry about what this value is. 01:12:22.172 --> 01:12:25.070 And it turns out it has no real impact 01:12:25.070 --> 01:12:28.171 on the asymptotic solution of the recurrence. 01:12:28.171 --> 01:12:32.037 How do we solve a recurrence like this? 01:12:32.037 --> 01:12:36.888 I now have T of n expressed in terms of T of n over 2. 01:12:36.888 --> 01:12:40.419 That's in the book and it is also in Lecture 2. 01:12:40.419 --> 01:12:43.901 We are going to do Lecture 2 to solve that, 01:12:43.901 --> 01:12:46.789 but in the meantime what I am going 01:12:46.789 --> 01:12:51.455 to do is give you a visual way of understanding what 01:12:51.455 --> 01:12:54.952 this costs, which is one of the techniques 01:12:54.952 --> 01:12:57.526 we will elaborate on next time. 01:12:57.526 --> 01:13:02.000 It is called a recursion tree technique. 01:13:02.000 --> 01:13:07.681 And I will use it for the actual recurrence that is almost 01:13:07.681 --> 01:13:12.494 the same 2T(n/2), but I am going to actually explicitly, 01:13:12.494 --> 01:13:17.249 because I want you to see where it occurs, 01:13:17.249 --> 01:13:23.048 plus some constant times n where c is a constant greater than 01:13:23.048 --> 01:13:23.548 zero. 01:13:23.548 --> 01:13:29.518 So, we are going to look at this recurrence with a base 01:13:29.518 --> 01:13:32.000 case of order one. 01:13:32.000 --> 01:13:34.672 I am just making the constant in here, 01:13:34.672 --> 01:13:37.340 the upper bound on the constant be 01:13:37.340 --> 01:13:39.322 explicit rather than implicit. 01:13:39.322 --> 01:13:43.092 And the way you do a recursion tree is the following. 01:13:43.092 --> 01:13:46.026 You start out by writing down the left hand 01:13:46.026 --> 01:13:47.345 side of the recurrence. 01:13:47.345 --> 01:13:51.562 And then what you do is you say well, that is equal to, 01:13:51.562 --> 01:13:53.907 and now let's write it as a tree. 01:13:53.907 --> 01:13:58.603 I do c of n work plus now I am going to have to do work 01:13:58.603 --> 01:14:01.000 on each of my two children. 01:14:01.000 --> 01:14:08.149 T of n over 2 and T of n over If I sum up what is in here, 01:14:08.149 --> 01:14:14.395 I get this because that is what the recurrence says, 01:14:14.395 --> 01:14:15.427 T(n)=2T(n/2)+cn. 01:14:15.427 --> 01:14:17.014 I have 2T(n/2)+cn. 01:14:17.014 --> 01:14:19.664 Then I do it again. 01:14:19.664 --> 01:14:21.496 I have cn here. 01:14:21.496 --> 01:14:23.786 I now have here cn/2. 01:14:23.786 --> 01:14:25.618 And here is cn/2. 01:14:25.618 --> 01:14:31.000 And each of these now has a T(n/4). 01:14:36.000 --> 01:14:39.972 And these each have a T(n/4). 01:14:39.972 --> 01:14:43.285 And this has a T(n/4). 01:14:43.285 --> 01:14:50.284 And I keep doing that, the dangerous dot, dot, dots. 01:14:50.284 --> 01:15:00.000 And, if I keep doing that, I end up with it looking like this. 01:15:18.000 --> 01:15:23.318 And I keep going down until I get to a leaf. 01:15:23.318 --> 01:15:27.896 And a leaf, I have essentially a T(1). 01:15:27.896 --> 01:15:29.702 That is T(1). 01:15:29.702 --> 01:15:35.608 And so the first question I ask here is, what 01:15:35.608 --> 01:15:38.983 is the height of this tree? 01:15:38.983 --> 01:15:39.489 Yeah. 01:15:39.489 --> 01:15:41.008 It's log n. 01:15:41.008 --> 01:15:45.880 It's actually very close to exactly log n 01:15:45.880 --> 01:15:51.144 because I am starting out at the top with n 01:15:51.144 --> 01:15:59.639 and then I go to n/2 and n/4 and all the way down until I 01:15:59.639 --> 01:16:04.324 get to The number of halvings of n until I get to 1 is 01:16:04.324 --> 01:16:07.070 log n so the height here is log n. 01:16:07.070 --> 01:16:10.076 It's OK if it is constant times log n. 01:16:10.076 --> 01:16:11.161 It doesn't matter. 01:16:11.161 --> 01:16:15.000 How many leaves are in this tree, by the way? 01:16:25.000 --> 01:16:28.328 How many leaves does this tree have? 01:16:28.328 --> 01:16:28.828 Yeah. 01:16:28.828 --> 01:16:31.951 The number of leaves, once again, 01:16:31.951 --> 01:16:34.238 is actually pretty close. 01:16:34.238 --> 01:16:35.438 It's actually n. 01:16:35.438 --> 01:16:38.847 If you took it all the way down. 01:16:38.847 --> 01:16:42.605 Let's make some simplifying assumption. n is 01:16:42.605 --> 01:16:47.849 a perfect power of 2, so it is an integer power of 2. 01:16:47.849 --> 01:16:52.235 Then this is exactly log n to get down to T(1). 01:16:52.235 --> 01:16:55.337 And then there are exactly n leaves, 01:16:55.337 --> 01:16:58.505 because the number of leaves here, 01:16:58.505 --> 01:17:05.000 the number of nodes at this level is 1, 2, 4, 8. 01:17:05.000 --> 01:17:08.627 And if I go down height h, I have 01:17:08.627 --> 01:17:13.963 2 to the h leaves, 2 to the log n, that is just n. 01:17:13.963 --> 01:17:17.172 We are doing math here, right? 01:17:17.172 --> 01:17:20.567 Now let's figure out how much work, 01:17:20.567 --> 01:17:25.511 if I look at adding up everything in this tree 01:17:25.511 --> 01:17:31.518 I am going to get T(n), so let's add that up. 01:17:31.518 --> 01:17:36.138 Well, let's add it up level by level. 01:17:36.138 --> 01:17:40.763 How much do we have in the first level? 01:17:40.763 --> 01:17:41.982 Just cn. 01:17:41.982 --> 01:17:47.786 If I add up the second level, how much do I have? 01:17:47.786 --> 01:17:48.285 cn. 01:17:48.285 --> 01:17:53.059 How about if I add up the third level? cn. 01:17:53.059 --> 01:17:57.443 How about if I add up all the leaves? 01:17:57.443 --> 01:17:58.989 Theta n. 01:17:58.989 --> 01:18:05.759 It is not necessarily cn because the boundary case 01:18:05.759 --> 01:18:09.397 may have a different constant. 01:18:09.397 --> 01:18:14.988 It is actually theta n, but cn all the way here. 01:18:14.988 --> 01:18:18.924 If I add up the total amount, that 01:18:18.924 --> 01:18:23.583 is equal to cn times log n, because that's 01:18:23.583 --> 01:18:31.233 the height, that is how many cn's I have here, plus theta n. 01:18:31.233 --> 01:18:39.258 And this is a higher order term than this, so this goes away, 01:18:39.258 --> 01:18:45.783 get rid of the constants, that is equal to theta(n lg n). 01:18:45.783 --> 01:18:51.674 And theta(n lg n) is asymptotically faster than 01:18:51.674 --> 01:18:52.786 theta(n^2). 01:18:52.786 --> 01:18:58.073 So, merge sort, on a large enough input size, 01:18:58.073 --> 01:19:03.000 is going to beat insertion sort. 01:19:03.000 --> 01:19:07.291 Merge sort is going to be a faster algorithm. 01:19:07.291 --> 01:19:10.579 Sorry, you guys, I didn't realize 01:19:10.579 --> 01:19:13.014 you couldn't see over there. 01:19:13.014 --> 01:19:16.682 You should speak up if you cannot see. 01:19:16.682 --> 01:19:22.662 So, this is a faster algorithm because theta(n lg n) grows 01:19:22.662 --> 01:19:25.048 more slowly than theta(n^2). 01:19:25.048 --> 01:19:31.000 And merge sort asymptotically beats insertion sort. 01:19:31.000 --> 01:19:35.424 Even if you ran insertion sort on a supercomputer, 01:19:35.424 --> 01:19:40.842 somebody running on a PC with merge sort for sufficient large 01:19:40.842 --> 01:19:46.441 input will clobber them because actually n^2 is way bigger than 01:19:46.441 --> 01:19:50.053 n log n once you get the n's to be large. 01:19:50.053 --> 01:19:55.572 And, in practice, merge sort tends to win here for n bigger 01:19:55.572 --> 01:19:58.000 than, say, 30 or so. 01:19:58.000 --> 01:20:02.092 If you have a very small input like 30 elements, 01:20:02.092 --> 01:20:06.272 insertion sort is a perfectly decent sort to use. 01:20:06.272 --> 01:20:11.497 But merge sort is going to be a lot faster even for something 01:20:11.497 --> 01:20:14.370 that is only a few dozen elements. 01:20:14.370 --> 01:20:18.289 It is going to actually be a faster algorithm. 01:20:18.289 --> 01:20:20.901 That's sort of the lessons, OK? 01:20:20.901 --> 01:20:24.786 Remember that to get your recitation assignments 01:20:24.786 --> 01:20:27.702 and attend recitation on Friday. 01:20:27.702 --> 01:20:33.952 Because we are going to be going through a bunch of the things 01:20:33.952 --> 01:20:36.871 that I have left on the table here. 01:20:36.871 --> 01:20:39.321 And see you next Monday.