Lecture 15: Dynamic Programming, Part 1: SRTBOT, Fib, DAGs, Bowling

Flash and JavaScript are required for this feature.

This is the first of four lectures on dynamic programing. This begins with how to solve a problem recursively and continues with three examples: Fibonacci, DAG shortest paths, and bowling.

Instructor: Erik Demaine

[SQUEAKING] [RUSTLING] [CLICKING]

ERIK DEMAINE: All right, welcome back to 006. Today we start a totally new section of the class. Up till now, we've mostly been showing you really cool and powerful algorithms, sorting algorithms, graph algorithms, data structures, trees, lots of good stuff that you can apply to solve tons of algorithmic problems, either by reducing to the data structures that we showed you, or reducing to graph problems that we showed you, or by modifying those algorithms a bit.

Today we're going to start a new section on algorithmic design-- how to, from scratch, come up with a polynomial time algorithm to solve a problem. And in particular, we're going to talk about a algorithmic design paradigm called dynamic programming, which is extremely powerful. It's probably the most powerful algorithmic design paradigm. Very general. Can solve lots of problems.

It's a particular type of recursive algorithm design. And in general, this class-- all of algorithms-- is about recursive algorithm design at some level, because we want to write constant-sized pieces of code that solve problems of arbitrary size. We have some problem size n and we're trying to write 100 lines of code or whatever, some constant amount that doesn't depend on the problem size. We have one algorithm that solves all instances of the problem.

And so we have to write code that is recursive or uses loops or somehow reuses the instructions that we give the computer. And you may know you can convert any algorithm based on loops into an algorithm using recursion. And we're going to take the recursive view today, in particular because it fits very well with our proof-by-induction technique, which we've used throughout this class, but also because it gives us some structure on how different subproblems relate in something called a subproblem graph, that we'll be talking about today.

And so we're going to start out with, in general, how do we design recursive algorithms? That's sort of the overall, encompassing everything. We have thought very hard to come up with a cool acronym for this paradigm which we invented called SRTBOT-- thanks, Jason. And so we'll talk-- it's not actually for sorting. It's just an acronym for sub-problems, relations, topological order, base case, original problem, and time.

But it's an acronym that will help you remember all the steps you need in order to specify a recursive algorithm. And dynamic programming is going to build on this template by adding one new idea called memoization, which is just the idea of reusing work that you've done before. And that's going to let us solve tons of problems.

And let's see. I don't-- let's get into it. So we'll start out today with SRTBOT. So here is SRTBOT down the column here. This is a recursive algorithm design paradigm. And in general, what we're going to do is take the problem that we actually want to solve and split it up into lots of possible sub problems.

And so the first part is to define what the heck are the subproblems. In general, we'll want some polynomial number of them. But it's pretty open-ended what these look like. And the hardest part, usually, in defining a recursive algorithm is figuring out what the sub problems should be. Usually they're related to the problem you want to solve.

Often the problem you want to solve-- this is actually near the last step-- the original problem you're trying to solve is often one of these sub problems. And then you use the smaller sub problems in order to build up the final, original problem. But sometimes at the end, you need to take a bunch of subproblems and combine them into your original problem.

You can think-- one analogy you can think of here is divide and conquer algorithms, which also had this kind of style. But more generally, we're going to relate different sub problem solutions with some recursive structure-- some recurrence relation. This is just a recursive algorithm that defines how to solve one problem in terms of smaller sub-problems for some notion of smaller.

And this is given by the topological order. So if we think of the subproblems as a graph and we draw an edge between-- so the vertices of the graph are sub problems. The edges are the dependencies between those subproblems. Then what we'd like is the topological ordering, the topological sort problem we talked about in the context of DFS or DAG shortest paths.

What we would like is that the subproblems and the calls-- the recursive calls between them in this recursive relation-- forms a DAG. We want it to be acyclic, otherwise you have an infinite loop in your recursive calls. If you have a cycle, you'll never terminate. And so to make sure that these dependencies between subproblems given by this recurrence relation is acyclic, one way to do that is to specify a topological order.

Or you could prove it some other way. But often it's just a for loop to say, just do it in this order. Then of course any recursive structure needs base cases. So that's a useful step not to forget. We want to solve the original problem using these sub problems. And then we analyze a running time at the end. So six easy steps. Actually, the hardest ones are these two, which are interrelated.

And what we're going to see over the next four lectures-- this is the first of four lectures on dynamic programming-- is lots of examples of applying this paradigm over and over together with the memoization idea, which we'll get to soon. Let's see an example first of an algorithm we've already seen, which is merge sort, so a divide and conquer algorithm, phrased with this structure of SRTBOT.

So for the sub problems-- so our original problem is to sort the elements of A. And some sub-problems that we solve along the way are sorting different sub-arrays of A. So for every-- well, not for every i and j, but for some i and js, we sort the items from i up to j minus 1. So I'm going to define that subproblem to be s of ij.

So this is something that I might want to solve. The original problem that I want to solve is s of 0 comma n, where n is the length of the array. So that's what I actually care about in the end. But we're going to solve that by writing it recursively in terms of sorting different sub-arrays as follows. This is the recurrence relation. I've written it very simply here.

Of course, there's a merge algorithm, which is somewhat complicated. But as we saw the two finger linear time merge algorithm, given two sorted arrays-- so this is supposed to be the sorted array version of the items i through m. m is the middle element between i and j and the sorted array of the items from m up to j.

If we merge those, that gives us the sorted array from i up to j. And that's exactly what merge sort does. So in general, this relation is just some algorithm for if you're given the solutions to some smaller subproblems, how do I solve the subproblem that I want to solve?

And so we need to make sure that this problem is bigger than the ones that we recursively call on and that we don't get an infinite cyclic loop of recursions. And here our valid topological order is to say, solve these problems in order where j minus i-- the length of the sub-array-- is increasing. And then you can check because m is strictly between i and j. As long as we're not in a base case, then we know we can-- these subarrays will be smaller than this one. And so this increasing order gives us a valid topological order on all of the problems, all the subproblems.

We have a base case, which is if we don't want to sort anything, that's the empty array, or at least in the original problem. And then running time is-- I mean, there's no better way to solve it than the recurrence that we already saw how to solve. So this is just another way to think of n log n merge sort in this labeled framework of SRTBOT.

Let's get to another problem that does not fit recursion so well. But we can make it better. So this is-- we're going to start with a very simple problem, which is computing Fibonacci numbers. It's really just a toy problem to illustrate a very powerful idea, which is memoization.

So the problem I'm interested in is I'm given a particular number, n. And I want to compute the nth Fibonacci number. And in case you forgot, the nth Fibonacci number is given by this recurrence. fn is fn minus 1 plus fn minus 2 with base case, let's say, f1 equals f2 equals 1.

And so we'd like to compute this. This seems-- this is a recurrence. So it seems very natural to write it as a recursive algorithm. So let's try to do it. We start with what are the sub problems. The obvious sub problems are just the various Fibonacci numbers, f i for i between 1 and n. So there are n of these sub problems.

Cool. Let's see. We want a relation between them. Well, maybe just to distinguish the problems from the Fibonacci numbers, let me write f of i. This is a function, an algorithm we're going to define. And it's defined to be-- the goal we're trying to get is the ith Fibonacci number given i.

And then we can write the recurrence relation on these guys, just f of i equals f of i minus 1 plus f of i minus 2. So in other words, recursively compute those Fibonacci numbers then add them together. That's an algorithm. Next is t for topological order.

Here, of course, we just want to compute these in order of increasing i from the base case is up. Another way I like to write this is as a for loop for i equals 1 to n. We will see why. But this gives an explicit order to compute these sub problems.

And base case is just the same as the Fibonacci numbers, but I guess I should write in parentheses. The original problem we want to solve is f of n. And the time-- all right, here's where things get interesting or bad. So what is the running time of this recursive algorithm? As I've stated it so far, the running time is given by a recurrence. Let's write the recurrence.

So in order to compute f of n, I recursively compute f of i minus 1 or f of n minus 1 here. And I recursively compute f of n minus 2. So that will take t of n minus 2. This first step will take t of n minus 1. And now I need to solve this recurrence.

This is not a recurrence that falls to the master method. It doesn't have a divided by. So we have to think about it a little bit. But we don't have to think about it too hard, because this recurrence is the same as this recurrence, which is the same as this recurrence. I've written it three times now. And so the solution to this is the nth Fibonacci number.

Oh, sorry. It's a little bit worse because in addition to those recursions, I also spend constant time to do the addition, maybe more than constant time. But if we just count the number of additions we do, it will be plus 1 additions. OK. But this is bigger than the nth Fibonacci number. And if you know anything about Fibonacci numbers, they grow exponentially. They're about golden ratio to the end. I'm wearing golden ratio, in case you forgot the number.

So that's bad, because golden ratio is bigger than 1. So this is exponential growth, as we know, especially in this time, exponential growth is bad. In algorithms, exponential growth is bad, because we can only solve very small problems with exponential growth. Very small n. So this is a terrible way to compute the nth Fibonacci number-- exponential bad.

OK, so don't do this. But there's a very tiny tweak to this algorithm that makes it really good, which is memoization. And this is a big idea. It is the big idea of dynamic programming. It's a funny word, probably made up by computer scientists. Instead of memorization, it's memoization, because we're going to write things down in a memo pad. It's the idea. And it's a very simple idea, which is just remember and reuse solutions to sub-problems.

So let's draw the recursion tree for this recursive algorithm as we've done it so far. So at the top, we-- let me make a little bit of space. At the top we are calling f of n. And then that calls f of n minus 1 and f of n minus 2. And it does an addition up here. And then this calls f of n minus 2. And this calls f of n minus 3.

This calls f of n minus 3. And this calls f of n minus 4. OK. And we notice that this sub problem is the same as this sub problem. So to compute f of n minus 1, I need f of minus 3. And also to compute f of n minus 2 I need f of n minus 3. So why are we computing it twice? Let's just do it once. When we solve it, let's write it in a table somewhere. And then when we need it again, we'll just reuse that value. Question?

AUDIENCE: What about the f of n minus 2?

ERIK DEMAINE: f of n minus 2 is also shared. So let me use a different symbol. f of n minus 2 is already here. So this was at the same level. But we also get shared reuse between different levels. In fact, I wouldn't even call f of n minus 3 because this whole part doesn't need to be computed a second time. If I already computed it here, it doesn't matter which one comes first. Let's say this one comes first.

Once this is done, I can write it down and reuse it over here. And then in here, we're going to call f of n minus three. So there's still another computation of f of n minus 3. When that one's done, I won't need to do this recursively. OK, so magically this is going to make this algorithm efficient with this very simple tweak. Let me write down the tweak more explicitly.

I won't write code here. But just describe it as a data structure. So we're going to maintain our good friend, the dictionary, which is abstract data type or interface. We could use different data structures to do it. But we're going to map some problems to their solutions, at least the ones that we've solved already.

And usually we can do this with just a direct access array, though you could use a hash table. Just get expected bounce. So when we write the code for our recursive function-- so in general, once we have a sort bot description, we can turn this into code. We define f of i. And it says am I in a base case? If so, return this. Otherwise, do this recursive call. That's our recursive algorithm.

But we're going to do a little more now. And first we're going to check whether this sub problem that we're trying to solve has already been solved. And if so, we return that storage solution. That's the easy case, but it might not exist.

And then we'll compute it in the usual way. So what the code then would look like to define f of i is first we check is i in our data structure. This is usually called the memo. So we say, is this sub-problem-- is i in my memo data structure? If so just return memo of i. Done. No recursion necessary.

Otherwise, check if I'm a base case. If so, done. Otherwise, recurse. So recursively call f of i minus 1 and f of i minus 2. And in this recursion, we can see that after we call f of i minus 1, in fact, it will have already computed f of i minus 2. So while this call is recursive, this one will immediately terminate because i minus 2 will already be in the memo table.

And so if you think about what happens, in fact, we'll just have recursion down the left branch of this thing. And all the right branches will be free. We can just look things up in the memo table. So what is the overall running time? For Fibonacci, this should be order n. Why is it order n? This is number of additions. Come back to that in a second.

In general, the way to analyze an algorithm like this that uses memoization is we just count how many different sub-problems are there? Because once we solve the sub-problem, we will never solve it again. That's the whole idea of a memo table. So we will solve each sub-problem at most once.

And so we just need to count, how much time does it take to solve every sub-problem? And here you can see it's constant. Either it's a base case and it takes constant time or we recursively call these things. But those are different sub-problems. So we're going to count those later.

And then the work that's actually done by this recurrence is a single addition. So in fact, it's n additions. To compute fn would be exactly n additions. So it turns out to be very nice closed form in this case. It should be exactly n sub problems to compute f of n because we started as dot at 1. And each one has one additional-- I guess not the base case. Maybe n minus 2.

OK. Definitely order n. Now, there's this one subtlety which-- let's forget about dynamic programming for a moment and go back to good old lecture one and two, talking about the word ram model of computation. A question here that usually doesn't matter in this class. Usually we assume additions take constant time. And we usually do that because it's usually true.

And in general, our model is the w bit additions-- where w is our machine word size-- takes constant time. But for this problem and this problem only, pretty much, for Fibonacci numbers, I happen to know that the Fibonacci numbers grow exponentially. So to write them down actually requires theta n bits because they are some constant to the n power. And so they're actually really big . n is probably bigger than w. Usually you think of problems that are much bigger than 64 or whatever your word size happens to be.

We do assume that w is at least log n. But n is probably bigger than w. It might be bigger or smaller. We don't know. And in general, to do an n bit addition-- these are n bit additions-- is going to take ceiling of n over w time. So in the end, we will spend this times n, because we have to do that, many of them, which is n plus n squared over w time.

So a bit of a weird running time. But it's polynomial, whereas this original recursive algorithm was exponential here. Using this one simple idea of just remembering the work we've done, suddenly this exponential time algorithm becomes polynomial. Why? Because we have few sub problems. We had n sub problems. And for each sub problem, we could write a recurrence relation that if we already knew the solutions to smaller sub problems, we could compute this bigger problem very efficiently.

This happened to be constant time or constant additions. n over w time. But as long as this is polynomial and this is polynomial, we're happy, because we have this nice formula that the time it takes is, at most, the sum over all sub problems of the relation time.

So I'm referring to sub problems, like a number of them and the time it takes to evaluate this, ignoring the recursive calls. That's important. This is the non recursive part. In the notes, I call this non-recursive work. So this formula gives us a way to bound the running time of one of these algorithms if we use memoization.

Without memoization, this is not true, Fibonacci to exponential time. But if we add memoization, we know that we only solve each sub-problem once. And so we just need to see, for each one, how much did it cost me to compute it, assuming all the recursion work is free, because that's already taken into account by the summation.

So in particular, this summation is at most the number of sub-problems times the time per sub-problem, which in this case was order n. We could try to apply that analysis to merge sort, because after all, this is also a recursive algorithm. It happens to not benefit from memoization. But we could throw in memoization. It wouldn't hurt us.

But if you think about the call graph here, which is like s of 0 m, which calls s of m-- 0 n over 2 and o of n over 2n and so on. It has the same picture, but there's actually no common substructure here. You'll never see a repeated sub-problem, because this range is completely disjoined from this range.

But you could throw in memoization and try to analyze in the same way and say, well, how many sub-problems are there? It looks like there's n choices for i and not quite n choices but it's at most n squared different choices. In fact, it's the triangular number sum of i equals 1 to n of i, different possible choices for inj.

But this is theta n squared sub-problems, which seems not so good. And then how much time are we spending per sub problem? Well, to solve s of ij, we have to merge about that many elements. We know merge takes linear time. And so this takes theta j minus i time to evaluate.

And so what we'd like to do is sum over all the sub problems of j minus i. This is the not triangular number but the tetrahedral number, I guess. And so we end up that the running time is, at most, n cubed. Great. So it's true that n log n is less than or equal to n cubed, but obviously not terribly useful. This algorithm by the way we already know how to analyze it is, indeed, n log n. And the running time turns out to be theta n log n.

So sometimes this equation is not what you want to use. But often it's good enough. And especially if you just want to get a polynomial upper bound, then you can try to optimize it later. This will give you a polynomial upper bound as long as the number of sub-problems is polynomial and the time per sub-problem is polynomial.

And indeed, n cubed is polynomial. It's not a great polynomial, but this is an alternate way to analyze merge sort. Obviously don't do this for merge sort. But it illustrates the technique. Good so far? Any questions? All right. Let me remember where we are. Cool.

So the next thing I'd like to do is show you one more algorithm that we've already seen in this class that fits very nicely into this structure-- arguably is a dynamic program-- and that is DAG shortest paths. So just to close the loop here, when I say dynamic programming, I mean recursion with memoization.

I mean, we take-- we write a recursive piece of code, which is like def f of some args, some sub-problem specification. We check is the problem in the memo table? If so, return memo of sub-problem.

And otherwise check if it's a base case and solve it if it's a base case. And otherwise, write the recurrence recurse via relation. And set the memo table of the sub-problem to be one of those things. OK, so this is the generic dynamic program. And implicitly, I'm writing Fibonacci in that way. And all of the dynamic programs have this implicit structure where I start with a memo table which is empty and I always just check if I'm in the memo table. If I am, I return it. Otherwise I compute according to this recursive relation by recursively calling f.

And that's it. So this is every DP algorithm is going to have that structure. And it's just using recursion and memoization together. OK, so now let's apply that technique to think about the DAG shortest paths problem. The problem was, I give you a DAG. I give you a source vertex, S-- single source shortest paths. Compute the shortest path weight from S to every vertex. That's the goal of the problem.

And we saw a way to solve that, which is DAG relaxation. I'm going to show you a different way, which turns out to be basically the same, but upside down, or flipped left right, depending which way you direct your edges. So what are our sub-problems? Well, here, actually, they're kind of spelled out for us. We want to compute delta and SV for all these. So that is size of these sub-problems.

That turns out to be enough for this overall problem. And the original problem we want to solve is all of the sub-problems. We solve all the sub-problems, we're done. And then we have-- I think we wrote this at some point during the DAG shortest paths lecture-- we have a recursive relation saying that the shortest way to get from s to v is the minimum of the shortest path to get to some vertex u plus the weight of the edge from u to v.

Why? Because if we look at a vertex v, unless we started there, we came from somewhere. And so we can consider all of the possible choices for the previous vertex u. And if you start at s and get to v, you must go through one of them. And so this is finding the best way among all the choices of u. What's the best way to get to u? And then take the edge from u to v for all edges uv.

And this is adjacency minus. We don't usually think of that. Usually we look at adjacency plus the outgoing edges. This is the incoming edges. And so u is an incoming-- uv is an incoming edge into v. OK, if we take that minimum-- and of course, possible there is no way to get to v. And so I'll also throw infinity into the set.

Take the min of that set. That will give me the shortest pathway in an acyclic graph from s to v. And great, this is recursive. This was a sub problem. These are sub problems which are smaller, I guess. There's no clear notion of smaller here, except we already know the clear notion of smaller is the topological order of our DAG.

Because our graph is acyclic, we know it has a topological order. We know how to compute it with DFS. And so that guarantees there's a topological order to compute these problems. And in fact, the relationship between problems is exactly the given graph, G. In order to compute the shortest pathway from s to v, I need to know the shortest pathway from s to all of the incoming vertices to v.

And so this is I guess in the call graph, this vertex calls this vertex, but direct the edge this way to say that this vertex requires-- this vertex needs to be computed before this one. And so then I can complete them in a topological order.

OK, we have a base case, which is delta of ss equals 0. And the running time is, again, we can use this formula and say, let's just sum over all the sub problems of the non recursive work in our recurrence relation and so it's computing this min. If I gave you these deltas for free and I gave you these weights, which we know from our weight data structure, how long does it take to compute this min? Well, however many things there are, however many numbers we're minning, which is the size of the incoming adjacency list plus 1 for that infinity.

And so if you compute this sum, sum of incoming edges to every vertex, that's all the edges. So this is v plus e. So in fact, this algorithm is morally the same algorithm as the one that we saw on the DAG shortest path lecture, which was compute a topological order and process vertices in that order and relax edges going out from vertices.

So here-- so in that algorithm, we would have tried to relax this edge if there was a better path to v. And the first one certainly is better than infinity. So the first one we relax indeed. The next edge, if this gave a better path from s to v, then we would relax that edge and update the way here and do the same here.

In the end, we're just computing this min in the relaxation algorithm but doing it step by step. In the relaxation algorithm, DAG relaxation, for each incoming edge to v, we update d of e if it's better. And so if you repeatedly update if you're better, that ends up computing a min. OK, so this is the same algorithm just kind of flipped backwards.

A funny thing, although we wrote down the topological order of the sub problem graph here is the topological order of g, because the sub-problem graph is g, the algorithm doesn't actually have to compute one. It's doing it automatically for free.

If you think about this algorithm, generic dp algorithm, which is check whether we're in a memo table. If so, return. Otherwise, recurse, or base case. This actually is a depth-first search through the sub-problem graph-- technically through the reverse of the sub-problem graph.

If I draw an edge-- so from small to big-- so I'm just saying, I orient the edges from my smaller sub-problems to the ones that need it-- then I'm actually depth-first searching backwards in this graph because the bigger problem calls the smaller problem. And the memo table is serving as the "have I visited this vertex already" check in DFS.

So this is actually a DFS algorithm. Plus we're doing some computation to actually solve the sub-problems we care about. So implicit in this algorithm, we are doing a DFS, and at the same time, we're doing this shortest path computation in the finishing order of that DFS traversal because all the edges are backwards. This is the same as the reverse finishing order if the graph is forwards.

So in the end, we're computing a topological order because dynamic programming includes in it depth first search. A lot of words. But it's kind of cool that this framework just solves DAG shortest paths without much work. I mean, we did a lot of work in shortest paths to prove that this relation is true. Once you know it's true, the algorithm part is pretty much free. You just write down SRTBOT and you're done.

OK. This brings us to in general-- at this point we have seen two examples of dynamic programming. I guess technically merge sort you could think of as a dynamic program, but it doesn't actually reuse anything. So it's not interesting. And indeed, that gave us a really bad bound.

We've definitely seen DAG shortest paths and Fibonacci numbers as two interesting examples. And what the next remainder of this lecture and the next three lectures are going to be about is more and more examples of dynamic programming and how you can use it to solve increasingly general problems.

So far, we've just solved an easy problem and a problem we already knew how to solve. Let's go to a new problem, which is bowling. Bowling is popular in Boston. Boston likes to play candlepin bowling, which is a bit unusual. Today we're going to play an even more unusual bowling game, one that I made up based on a bowling game that Henry [INAUDIBLE] made up in 1908. So ancient bowling, I'll call it, or I think linear bowling is what I might call it. I'll just call it bowling here.

And now I'm going to attempt to draw a bowling pin. Not bad. They might get progressively worse. So imagine n identical bowling pins. Please pretend these are identical. And I have a ball which is approximately the same size as a bowling pin. These bowling pins are pretty close together. I should have left a little gap here.

And you are a really good bowler. Now, unfortunately, these bowling pins are on a line. And you're bowling from way down at infinity. So when you bowl, you can only hit one pin or two pins or zero pins. But probably you want to hit some pins. So if you bowl straight out of pin, you will just hit that one pin. And if you bowl in the middle between two pins, you will knock down-- that's a ball, sorry-- you will knock down two pins. And this is your model of bowling, model of computation.

Now, what makes this interesting is that the pins have values. Pin i has value-- this is obviously a toy problem, though this problem-- this type of bowling does go back to 1908, it was also a toy problem in that setting. So each of these bowling pins has some number on it, let's say 1, 9, 9-- I'll do a slightly more interesting example, maybe another one here and a 2 and a 5 and a 5, something like this.

OK. Or maybe make it a little more interesting. Let's put some negative numbers on here. OK. And the model-- so you're at the carnival bowling. Each pin has different-- potentially different values. And the model is if you hit one pin, i, then you get vi points. So that's straight forward. To make it interesting, when you hit two pins, you get the product.

So if I hit two pins, it's always i and i plus 1 for some I. You get vi times vi plus 1 points. This is the game you're playing. And it doesn't really matter that this is a product. Product is just some weird function that's hard to imagine.

If you stare at this long enough, you should convince yourself that the optimal solution is probably to-- so, for each of these numbers, I could leave it singleton or pair it with its left neighbor or pair it with its right neighbor. But the pairings can't overlap because once I hit a pin, it's gone. It's knocked over. It disappears.

So because of these nine, which are a very high value, what I'd probably like to do is hit both of them together, so pair them up, because 9 times 9 is 81. That's really big, much better than hitting them individually or hitting 9 times 1 or 9 times 2.

1 and 1 is kind of funny, because it's actually better to hit them individually. That will give you two points, whereas if I'd pair them up, I only get one point. 2 and minus 5, that seems bad. Negative 10 points. My goal is to maximize score.

Do you have to hit all the pins? Let's say no, you don't have to hit all the pins. So I could skip the minus fives. But in fact, here, because they're adjacent, minus 5 times minus 5 is good. That's 25 points. So the optimal solution for this particular instance are to hit all the pins, these positive, these together, these together. If I added, for example, another pin of minus 3 here, I would choose not to hit that pin. Good question.

So you just play until you are tired. When you decide to stop playing, how can I maximize your score? There are many variations of this game. All of them-- basically any variation-- not literally every variation, but many, many variations of this problem can all be solved quickly with dynamic programming. But let's solve this particular one. OK.

So now we're really in algorithmic design mode. We need to think about SRTBOT. And in particular, we need to think about what would the sub problems be here? And at this point, we don't have a lot of help. So I should probably give you some tools. If I want to solve a problem like this, the input is a sequence of numbers. It's a sequenced data structure. Maybe it's an array of numbers, which is this v array.

And let's see. A general tool for sub-problem design which will cover most of the problems-- maybe all of the problems that we see in this class for dynamic programming. Here's a trick. If your input is a sequence, here are some good sub-problems to consider.

We could do all prefixes. So let's call the sequence x. So we could do x prefix means up to a given i for all i. We could do all the suffixes, x from i onward for all i. Or we could do substrings, which are the consecutive items from i to j. I don't write subsequence here. Subsequence means you can omit items in the middle.

So substring you have to start in some position and do all the things up to j. So these are nice, easy to express in Python notation. And these are great, because they're polynomial. If I have n things-- if the length of my sequence, x, is n, then there are n prefixes-- technically n plus 1. So let's do theta n prefixes. There are theta n suffixes. And there are theta n squared substrings because there's n-- roughly n choices for i and j separately. Sorry?

Sub-sequences. Good. Right. I didn't write sub-sequences, because in fact, there are exponentially many sub sequences. It's 2 to the n. For every item, I could choose it or not. So I don't want to parameterize-- I don't want my sub problems to be sub sequences because that's guaranteed-- well, then you're guaranteed to get an exponential number of sub-problems, which is bad.

We'd like to balance the numbers of sub-problems by polynomial. So these are three natural ways to get polynomial bounds. Now, prefixes and suffixes are obviously better because there's fewer of them, linear instead of quadratic. And usually almost every problem you encounter, prefixes and suffixes are equally good. It doesn't really matter which one you choose.

So maybe you'd like to think of-- well, we'll get to-- just choose whichever is more comfortable for you. But sometimes it's not enough. And we'll have to go to substrings. That won't be for another lecture or two. Today I claim that prefixes or suffixes are enough to solve the bowling problem.

So what we're going to do is think about-- I prefer suffixes usually, because I like to work from left to right, from the beginning to the end. So we're going to think of a suffix of the bowling pins. And so what is the sub-problem on a suffix? Well, a natural version is just to solve the original problem, bowling. How do I maximize my score if all I were given were these pins?

Suppose the pins to the left of i didn't exist. How would I maximize my score on the remaining pins? Or for this suffix, given these four pins, what would I do? And there's some weird sub problems here. If I just gave you the last pin, what would you do? Nothing. That's clearly different from what I would do globally here.

But I claim if I can solve all suffixes I can solve my original problem, because one of the suffixes is the whole sequence. So let's do it. Sort by for bowling. So here is our dynamic program. The sub-problems are suffixes. So I'll write b of i is the maximum score we could get possible with our starting-- if we started a game with pins i, i plus 1, up to n minus 1, which is a suffix of the pins.

Very important whenever you write a dynamic program to define what your sub-problems are. Don't just say how to compute them, but first say what is the goal of the sub problem. This is a common mistake to forget to state what you're trying to do. So now I have defined b of i. Now, what is the original thing I'm trying to solve?

You also put in SRTBOT-- you could put the O earlier, then it actually spells sort. So why don't I do that for fun. The original problem we're trying to solve is b of 0, because that is all of the pins. The suffix starting at 0 is everything. So if we can solve that, we're done. Next is r for relate. This is the test of, did I get the sub-problems right, is whether I can write a recurrence relation.

So let's try to do it. We want to compute b of i. So we have pin i here and then the remaining pins. And the big idea here is to just think about-- the nice thing about suffixes is if I take off something from the beginning, I still have a suffix.

Remember, my goal is to take this sub-problem, which is suffix starting at i, and reduce it to a smaller sub problem, which means a smaller suffix. So I'd like to clip off one or two items here. And then the remaining problem will be one of my sub problems. I'll be able to recursively call b of something smaller than i-- or sorry, b of something larger than i will be a smaller subsequence because we're starting later.

OK, so what could I do? Well, the idea is to just look at pin i and think, well, what could I do to pin i? I could not hit it ever with a ball. I could skip it. That's one option. What would be my score then? Well, if I skip pin i, that leaves the remaining pins, which is just a smaller suffix. So that is b of i plus 1.

I'm going to write a max out here because I'd like to maximize my score. And one of the options is, forget about pin i. Just solve the rest. Another option is I throw a ball. And I exactly hit pin i. That's one thing I could do. And it would leave exactly the same remainder. So another option is b of i plus 1 plus vi.

Why would I prefer this over this? Well, if vi is negative, I'd prefer this. But if vi is positive, I'd actually prefer this over that. So you can figure out which is better, just locally. But then there's another thing I can do, which is maybe I hit this pin in a pair with some other pin. Now, there's no pin to the left of this one. We're assuming we only have the suffix. And so the only other thing I can do is throw a ball and hit i together with i plus 1. And then I get the product.

Now, what pins remain? i plus 2 on. Still a suffix. So if I remove one or two items, of course, I still get a suffix-- in this case, b of i plus 2-- and then the number of points that I add on are vi times vi plus 1. So this is a max of three things. So how long does it take me to compute it? I claim constant time.

If I don't count the time it takes to compute these other sub problems, which are smaller because they are smaller suffixes further to the right, then I'm doing a couple of additions-- product, max. These are all nice numbers and I'll assume that they live in the w-bit word, because we're only doing constant sized products. That's good.

So this takes constant, constant non-recursive work. How many sub problems are? Well, it's suffixes, so it's a linear number of sub problems. And so the time I'm going to end up needing is number of sub problems, n, times the non-recursive work I do per sub problem, which is constant. And so this is linear time. Great.

And I didn't finish SRTBOT, so there's another t, which is to make sure that there is a topological order and that is in decreasing i order. Or I might write that as a for loop-- for i equals n, n minus 1. This is the order that I would compute my problems because the suffix starting at n is the empty suffix. The suffix starting at 0, that's the one I actually want to compute. That's the final suffix I should be computing.

And then we have a b for base case, which is that first case, b of n equals 0, because there's no pins. So I don't get any points. Sad. OK, so this is it. We just take these components, plug them into this recursive, memoized algorithm, and we have a linear time algorithm. I want to briefly mention a different way you could plug together those pieces, which is called bottom up dp, which is-- let's do it for this example.

So if I have-- let's see. Let me start with the base case, b of n equals 0. But now it's an assignment. And I'm going to do for loop from the topological order for i equals n, n minus 1 to 0. Now I'm going to do the relation, b of i equals max of b of i plus 1 and b of i plus 1 plus bi and b of i plus 2 plus di vi plus 1. Technically this only works if i is strictly less than n minus 1.

So I should have an if i is less than minus 1 for that last part because I can only do-- I can only hit two pins if there's at least two pins left. And then return b of 0. So what I just did is a transformation from this SRTBOT template into a non-recursive algorithm, a for loop algorithm, where I wrote my base case first. Then I did my topological order. Then I did my relation.

Then at the end, I did my-- not base case. The original problem. And provided you can write your topological order as some for loops. This is actually a great way to write down a dp as code. If I were going to implement this algorithm, I would write it this way, because this is super fast. No recursive calls. Just one for loop.

In fact, this is almost a trivial algorithm. It's amazing that this solves the bowling problem. It's in some sense considering every possible strategy I could for bowling these pins. What we're using is what we like to call local brute force, where when we think about pin i, we look at all of the possible things I could do to pin i, here there's really only three options of what I could do.

Now, normally, if I tried all the options for pin i and then all the options for i plus 1 and i plus 2 and so on, that would be exponential. It'd be 3 times 3 times 3. That's bad, but because I can reuse these sub problems, it turns out to only be linear time. It's almost like magic. dp-- dp is essentially an idea of using local brute force.

And by defining a small number of sub-problems up front-- and as long as I stay within those sub problems, as long as I'm always recursing into this polynomial space, I end up only doing polynomial work, even though I'm in some sense exploring exponentially many options. And it is because what I do to this pin doesn't depend too much to what I do to a pin much later.

There's a lot of intuition going on here for what-- when DP works. But we're going to see a lot more examples of that coming up. And I just want to mention the intuition for how to write a recurrence like this is to think about-- in the case of suffixes, you always want to think about the first item, or maybe the first couple of items. The case of prefixes, you always think about the last item. And for substrings, it could be any item-- maybe in the middle.

If I remove an item from the middle of a substring, I get two substrings, so I can recurse. Here or in general, what we want to do is identify some feature of the solution that if we knew that feature we would be done. We would reduce to a smaller sub problem. In this case, we just say, well, what are the possible things I could do to the first pin?

There are three options. If I knew which option it was, I would be done. I could recurse and do my addition. Now, I don't know which thing I want to do. So I just try them all and take the max. And if you're maximizing, you take the max. If you're minimizing, you take the min. Sometimes you take an or or an and. There might be some combination function.

For optimization problems where you're trying to maximize or minimize something, like shortest paths you're trying to minimize, we put them in here. So usually it's min or max. And this is extremely powerful. All you need to do-- the hard part is this inspired design part where you say, what do I need to know that would let me solve my problem?

And if you can identify that and the number of choices for what you need to know is polynomial, then you will be able to get a polynomial dynamic program. That's the intuition. You'll see a lot more examples in the next three lectures.