Lecture 12: Skip Lists

Related Resources

Lecture Notes (PDF)
Assignments
Exams

• Download this transcript - PDF (English - US)

Good morning. Today we're going to talk about it a balanced search structure, so a data structure that maintains a dynamic set subject to insertion, deletion, and search called skip lists.

So, I'll call this a dynamic search structure because it's a data structure. It supports search, and it's dynamic, meaning insert and delete.

So, what other dynamic search structures do we know,

just for sake of comparison, and to wake everyone up?

Shut them out, efficient, I should say,

also good, logarithmic time per operation.

So, this is a really easy question to get us off the ground. You've seen them all in the last week, so it shouldn't be so hard.

Treap, good. On the problems that we saw treaps. That's, in some sense, the simplest dynamic search structure you can get from first principles because all we needed was a bound on a randomly constructed binary search tree. And then treaps did well.

So, that was sort of the first one you saw depending on when you did your problem set. What else?

Charles? Red black trees, good answer. So, that was exactly one week ago. I hope you still remember it.

They have guaranteed log n performance.

So, this was an expected bound. This was a worst-case order log n per operation, insert, delete, and search. And, there was one more for those who want to recitation on Friday: B trees, good. And, by B trees, I also include two-three trees, two-three-four trees, and all those guys. So, if B is a constant, or if you want your B trees knows a little bit cleverly, that these have guaranteed order log n performance, so, worst case, order log n.

So, you should know this. These are all balanced search structures. They are dynamic.

They support insertions and deletions.

They support searches, finding a given key.

And if you don't find the key, you find its predecessor and successor pretty easily in all of these structures.

If you want to augment some data structure, you should think about which one of these is easiest to augment, as in Monday's lecture. So, the question I want to pose to you is supposed I gave you all a laptop right now, which would be great. Then I asked you, in order to keep this laptop you have to implement one of these data structures, let's say, within this class hour. Do you think you could do it?

How many people think you could do it?

A couple people, a few people, OK, all front row people, good.

I could probably do it. My preference would be B trees.

They're sort of the simplest in my mind.

This is without using the textbook.

This would be a closed book exam.

I don't have enough laptops to do it, unfortunately.

So, B trees are pretty reasonable.

Deletion, you have to remember stealing from a sibling and whatnot. So, deletions are a bit tricky.

Red black trees, I can never remember it.

I'd have to look it up, or re-derive the three cases.

treaps are a bit fancy. So, that would take a little while to remember exactly how those work.

You'd have to solve your problem set again, if you don't have it memorized. Skip lists, on the other hand, are a data structure you will never forget, and something you can implement within an hour, no problem. I've made this claim a couple times before, and I always felt bad because I had never actually done it. So, this morning, I implemented skip lists, and it took me ten minutes to implement a linked list, and 30 minutes to implement skip lists. And another 30 minutes debugging them. There you go.

It can be done. Skip lists are really simple.

And, at no point writing the code did I have to think, whereas every other structure I would have to think.

There was one moment when I thought, ah, how do I flip a coin? That was the entire amount of thinking. So, skip lists are a randomized structure. Let's add in another adjective here, and let's also add in simple.

So, we have a simple, efficient, dynamic, randomized search structure: all those things together.

So, it's sort of like treaps and that the bound is only a randomized bound. But today, we're going to see a much stronger bound than an expectation bound.

So, in particular, skip lists will run in order log n expected time. So, the running time for each operation will be order log n in expectation.

But, we're going to prove a much stronger result that their order log n, with high probability.

So, this is a very strong claim.

And it means that the running time of each operation, the running time of every operation is order log n almost always in a certain sense. Why don't I foreshadow that?

So, it's something like, the probability that it's order log n is at least one minus one over some polynomial, and n. And, you get to set the polynomial however large you like.

So, what this basically means is that almost all the time, you take your skip lists, you do a polynomial number of operations on it, because presumably you are running a polynomial time algorithm that using this data structure. Do polynomial numbers of inserts, delete searches, every single one of them will take order log n time, almost guaranteed.

So this is a really strong bound on the tail of the distribution. The mean is order log n.

That's not so exciting. But, in fact, almost all of the weight of this probability distribution is right around the log n, just tiny little epsilons, very tiny probabilities you could be bigger than log n.

So that's where we are going. This is a data structure by Pugh] in 1989. This is the most recent.

Actually, no, sorry, treaps are more recent.

They were like '93 or so, but a fairly recent data structure for just insert, delete, search.

And, it's very simple. You can derive it if you don't know anything about data structures, well, almost nothing. Now, analyzing that the performance is log n, that, of course, takes our sophistication. But the data structure itself is very simple. We're going to start from scratch. Suppose you don't know what a red black tree is. You don't know what a B tree is. Suppose you don't even know what a tree is. What is the simplest data structure for storing a bunch of items for storing a dynamic set?

A list, good, a linked list.

Now, suppose that it's a sorted linked list.

So, I'm going to be a little bit fancier there.

So, if you have a linked list of items, here it is, maybe we'll make it doubly linked just for kicks, how long does it take to search in a sorted linked list?

Log n is one answer. n is the other answer.

Which one is right? n is the right answer.

So, even though it's sorted, we can't do binary search because we don't have random-access into a linked list. So, suppose I'm only given a pointer to the head. Otherwise, I'm assuming it's an array. So, in a sorted array you can search in log n. Sorted linked list: you've still got to scan through the darn thing.

So, theta n, worst case search.

Not so good, but if we just try to improve it a little bit, we will discover skip lists automatically. So, this is our starting point: sorted linked lists, data n time.

And, I'm not going to think too much about insertions and deletions for the moment. Let's just get search better, and then we'll worry about dates.

Updates are where randomization will come in.

Search: pretty easy idea. So, how can we make a linked list better? Suppose all we know about our linked lists. What can I do to make it faster? This is where you need a little bit of innovation, some creativity.

More links: that's a good idea. So, I do try to maybe add pointers to go a couple steps ahead.

If I had log n pointers, I could do all powers of two ahead. That's a pretty good search structure. Some people use that; like, some peer-to-peer networks use that idea.

But that's a little too fancy for me.

Ah, good. You could try to build a tree on this linear structure. That's essentially where we're going. So, you could try to put pointers to, like, the middle of the list from the roots. So, you search between either here. You point to the median, so you can compare against the median, and know whether you should go in the first half or the second half that's definitely on the right track, also a bit too sophisticated.

Another list: yes.

Yes, good. So, we are going to use two lists. That's sort of the next simplest thing you could do. OK, and as you suggested, we could maybe have pointers between them.

So, maybe we have some elements down here, some of the elements up here. We want to have pointers between the lists. OK, it gets a little bit crazy in how exactly you might do that.

But somehow, this feels good.

So this is one linked list: L_1.

This is another linked list: L_2.

And, to give you some inspiration, I want to give you, so let's play a game. The game is, what is this sequence? So, the sequence is 14.

If you know the answer, shout it out.

Anyone yet? OK, it's tricky.

It's a bit of a small class, so I hope someone knows the answer.

How many TA's know the answer? Just a couple, OK, if you're looking at the slides, probably you know the answer. That's cheating.

OK, I'll give you a hint. It is not a mathematical sequence. This is a real-life sequence.

Yeah? Yeah, and what city?

New York, yeah, this is the 7th Ave line.

This is my favorite subway line in New York.

But, what's a cool feature of the New York City subway?

OK, it's a skip list. Good answer.

[LAUGHTER] Indeed it is. Skip lists are so practical.

They've been implemented in the subway system.

How cool is that? OK, Boston subway is pretty cool because it's the oldest subway definitely in the United States, maybe in the world. New York is close, and it has other nice features like it's open 24 hours.

That's a definite plus, but it also has this feature of express lines. So, it's a bit of an abstraction, but the 7th Ave line has essentially two kinds of cars. These are street numbers by the way. This is, Penn Station, Times Square, and so on.

So, there are essentially two lines.

There's the express line which goes 14, to 34, to 42, to 72, to 96.

And then, there's the local line which stops at every stop.

And, they accomplish this with four sets of tracks.

So, I mean, the express lines have their own dedicated track.

If you want to go to stop 59 from, let's say, Penn Station, well, let's say from lower west side, you get on the express line.

You jump to 42 pretty quickly, and then you switch over to the local line, and go on to 59 or wherever I said I was going.

OK, so this is express and local lines, and we can represent that with a couple of lists.

We have one list, sure, we have one list on the bottom, so leave some space up here.

This is the local line, L_2, 34, 42, 50, 59, 66, 72, 79, and so on.

And then we had the express line on top, which only stops at 14, 34, 42, 72, and so on.

I'm not going to redraw the whole list.

You get the idea. And so, what we're going to do is put links between in the local and express lines, wherever they happen to meet. And, that's our two linked list structure. So, that's what I actually meant what I was trying to draw some picture.

Now, this has a property that in one list, the bottom list, every element occurs. And the top list just copies some of those elements. And we're going to preserve that property. So, L_2 stores all the elements, and L_1 stores some subset.

And, it's still open which ones we should store.

That's the one thing we need to think about.

But, our inspiration is from the New York subway system.

OK, there, that the idea. Of course, we're also going to use more than two lists. OK, we also have links.

Let's say it links between equal keys in L_1 and L_2.

Good. So, just for the sake of completeness, and because we will need this later, let's talk about searches before we worry about how these lists are actually constructed. Of course, if I wanted that board. So, if you want to search for an element, x, what do you do?

Well, this is the taking the subway algorithm.

And, suppose you always start in the upper left corner of the subway system, if you're always in the lower west side, 14th St, and I don't know exactly where that is, but more or less, somewhere down at the bottom of Manhattan. And, you want to go to a particular station like 59. Well, you'd stay on the express line as long as you can because it happens that we started on the express line. And then, you go down.

And then you take the local line the rest of the way.

That's clearly the right thing to do if you always start in the top left corner. So, I'm going to write that down in some kind of an algorithm because we will be generalizing it. It's pretty obvious at this point. It will remain obvious.

So, I want to walk right in the top list until that would go too far. So, you imagine giving someone directions on the subway system they've never been on.

So, you say, OK, you start at 14th.

Take the express line, and when you get to 72nd, you've gone too far. Go back one, and then go down to the local line.

It's really annoying directions.

But this is what an algorithm has to do because it's never taken the subway before. So, it's going to check, so let's do it here. So, suppose I'm aiming for 59.

So, I started 14, say the first thing I do is go to 34. Then from there, I go to 42. Still good because 59 is bigger than 42. I go right again.

I say, oops, 72 is too big.

That was too far. So, I go back to where it just was. Then I go down and then I keep going right until I find the element that I want, or discover that it's not in the bottom list because bottom list has everyone. So, that's the algorithm.

Stop when going right would go too far, and you discover that with a comparison. Then you walk down to L_2.

And then you walk right in L_2 until you find x, or you find something greater than x, in which case x is definitely not on your list. And you found the predecessor and successor, which may be your goal.

If you didn't find where x was, you should find where it would go if it were there, because then maybe you could insert there. We're going to use this algorithm in insertion. OK, but that search: pretty easy at this point. Now, what we haven't discussed is how fast the search algorithm is, and it depends, of course, which elements we're going to store in L_1, which subset of elements should go in L_1.

Now, in the subway system, you probably put all the popular stations in L_1. But here, we want worst-case performance. So, we don't have some probability distribution on the nodes.

We just like every node to be accessed sort of as quickly as possible, uniformly. So, we want to minimize the maximum time over all queries. So, any ideas what we should do with L_1? Should I put all the nodes of L_1 in the beginning? OK, it's a strict subset.

Suppose I told you what the size of L_1 was.

I can tell you, I could afford to build this many express stops. How should you distribute them among the elements of L_2? Uniformly, good.

So, what nodes, sorry, what keys, let's say, go in L_1? Well, definitely the best thing to do is to spread them out uniformly, OK, which is definitely not what the 7th Ave line looks like.

But, let's imagine that we could reengineer everything.

So, we're going to try to space these things out a little bit more. So, 34 and 42nd are way too close. We'll take a few more stops.

And, now we can start to analyze things.

OK, as a function of the length of L_1.

So, the cost of a search is now roughly, so, I want a function of the length of L_1, and the length of L_2, which is all the elements, n.

What is the cost of the search if I spread out all the elements in L_1 uniformly? Yeah?

Right, the total number of elements in the top lists, plus the division between the bottom and the top.

So, I'll write the length of L_1 plus the length of L_2 divided by the length of L_1. OK, this is roughly, I mean, there's maybe a plus one or so here because in the worst case, I have to search through all of L_1 because the station I could be looking for could be the max.

OK, and maybe I'm not lucky, and the max is not on the express line. So then, I have to go down to the local line. And how many stops will I have to go on the local line? Well, L_1 just evenly partitions L_2. So this is the number of consecutive stations between two express stops.

So, I take the express, possibly this long, but I take the local possibly this long.

And, this is an L_2. And there is, plus, a constant, for example, go walking down. But that's basically the number of nodes that I visit. So, I'd like to minimize this function. Now, L_2, I'm going to call that n because that's the total number of elements.

L_1, I can choose to be whatever I want.

So, let's go over here. So, I want to minimize L_1 plus n over L_1. And I get to choose L_1.

Now, I could differentiate this, set it to zero, and go crazy. Or, I could realize that, I mean, that's not hard. But, that's a little bit too fancy for me. So, I could say, well, this is clearly best when L_1 is small.

And this is clearly best when L_1 is large.

So, there's a trade-off there. And, the trade-off will be roughly minimized up to constant factors when these two terms are equal. That's when I have pretty good balance between the two ends of the trade-off.

So, this is up to constant factors.

I can let L_1 equal n over L_1, OK, because at most I'm losing a factor of two there when they happen to be equal.

So now, I just solve this. This is really easy.

This is (L_1)^2 equals n. So, L_1 is the square root of n. OK, so the cost that I'm getting over here, L_1 plus L_2 over L_1 is the square root of n plus n over root n, which is, again, root n. So, I get two root n.

So, search cost, and I'm caring about the constant here, because it will matter in a moment. Two square root of n: I'm not caring about the additive constant, but the multiplicative constant I care about.

OK, that seems good. We started with a linked list that searched in n time, theta n time per operation.

Now we have two linked lists, search and theta root n time.

It seems pretty good. This is what the structure looks like. We have root n guys here.

This is in the local line. And, we have one express stop which represents that. But we have another root n values in the local line. And we have one express stop that represents that. And these two are linked, and so on.

Well, I should put some dot, dot, dots in there.

OK, so each of these chunks has length root n, and the number of representatives up here is square root of n. The number of express stops is square root of n. So clearly, things are balanced now. I search for, at most, square root of n up here.

Then I search in one of these lists for, at most, square root of n. So, every search takes, at most, two root n. Cool, what should we do next?

So, again, ignore insertions and deletions.

I want to make searches faster because square root of n is not so hot as we know. Sorry?

More lines. Let's add a super express line, or another linked list. OK, this was two.

Why not do three? So, we started with a sorted linked list. Then we went to two.

This gave us two square root of n.

Now, I want three sorted linked lists.

I didn't pluralize here. Any guesses what the running time might be? This is just guesswork.

Don't think. From two square root of n, you would go to, sorry?

Two square root of two, fourth root of n?

That's on the right track. Both the constant and the root change, but not quite so fancily.

Three times the cubed root: good.

Intuition is very helpful here. It doesn't matter what the right answer is. Use your intuition.

You can prove that. It's not so hard.

You now have three lists, and what you want to balance are at the length of the top list, the ratio between the top two lists, and the ratio between the bottom two lists.

So, you want these three to multiply out to n, because the top times the ratio times the ratio: that has to equal n. And, so that's where you get the cubed root of n. Each of these should be equal.

So, you set them because the cost is the sum of those three things. So, you set each of them to cubed root of n, and there are three of them.

OK, check it at home if you want to be more sure.

Obviously, we want a few more. So, let's think about k sorted lists. k sorted lists will be k times the k'th root of n. You probably guessed that by now. So, what should we set k to?

I don't want the exact minimum. What's a good value for k?

Should I set it to n? n's kind of nice, because the n'th root of n is just one.

Now that's n. So, this is why I cared about the lead constant because it's going to grow as I add more lists. What's the biggest reasonable value of k that I could use? Log n, because I have a k out there. I certainly don't want to use more than log n. So, log n times the log n'th root, and this is a little hard to draw of n.

Now, what is the log n'th root of n?

That's what you're all thinking about.

What is the log n'th root of n minus two?

It's one of these good questions whose answer is?

Oh man. Remember the definition of root? OK, the root is n to the one over log n. OK, good, remember the definition of having a power, A to the B?

It was like two to the power, B log A?

Does that sound familiar? So, this is two to the log n over log n, which is, I hope you can get it at this point, two. Wow, so the log n'th root of n minus two is zero: my favorite answer.

OK, this is to. So this whole thing is two log n: pretty nifty. So, you could be a little fancier and tweak this a little bit, but two log n is plenty good for me. We clearly don't want to use any more lists, but log n lists sounds pretty good. I get, now, logarithmic search time. Let's check.

I mean, we sort of did this all intuitively.

Let's draw what the list looks like.

But, it will work. So, I'm going to redraw this example because you have to, also.

So, let's redesign that New York City subway system.

And, I want you to leave three blank lines up here.

So, you should have this memorized by now.

But I don't. So, we are not allowed to change the local line, though it would be nice, add a few more stops there. OK, we can stop at 79th Street.

That's enough. So now, we have log n lists.

And here, log n is about four. So, I want to make a bunch of lists here. In particular, 14 will appear on all of them. So, why don't I draw those in?

And, the question is, which elements go in here?

So, I have log n lists. And, my goal is to balance the number of items up here, and the ratio between these two lists, and the ratio between these two lists, and the ratio between these two lists.

I want all these things to be balanced.

There are log n of them. So, the product of all those ratios better be n, the number of elements down here. So, the product of all these ratios is n. And there's log n of them; how big is each ratio? So, I'll call the ratio r.

The ratio's r. I should have r to the power of log n equals n. What's r?

What's r minus two? Zero.

OK, this should be two to the power of log n.

So, if the ratio between the number of elements here and here is to all the way down, then I will have an elements at the bottom, which is what I want.

So, in other words, I want half the elements here, a quarter of the elements here, an eighth of the elements here, and so on. So, I'm going to take half of the elements evenly spaced out: 34th, 50th, 66th, 79th, and so on. So, this is our new semi-express line: not terribly fast, but you save a factor of two for going up there.

And, when you're done, you go down, and you walk, at most, one step.

And you find what you're looking for.

OK, and then we do the same thing over and over and over until we run out of elements. I can't read my own writing.

It's 79th.

OK, if I had a bigger example, I would be more levels, but this is just barely enough. Let's say two elements is where I stop. So, this looks good.

Does this look like a structure you've seen before, at all, vaguely? Yes?

A tree: yes. It looks a lot like a binary tree. I'll just leave it at that.

In your problem set, you'll understand why skip lists are really like trees. But it's more or less a tree.

Let's say at this level, it looks sort of like binary search. You look at 14; you look at 15, and therefore, you decide whether you are in the left half for the right half. And that's sort of like a tree.

It's not quite a tree because we have this element repeated all over. But more or less, this is a binary tree. At depth I, we have two to the I nodes, just like a tree, just like a balanced tree.

I'm going to call this structure an ideal skip list.

And, if all we are doing our searches, ideal skip lists are pretty good. Maybe at practice: not quite as good as a binary search tree, but up to constant factors: just as good. So, for example, I mean, we can generalize search, just check that it's log n. So, the search procedure is you start at the top left. So, let's say we are looking for 72. You start at the top left.

14 is smaller than 72, so I try to go right.

79 is too big. So, I follow this arrow, but I say, oops, that's too much.

So, instead, I go down 14 still.

I go to the right: oh, 50, that's still smaller than 72: OK. I tried to go right again.

Oh: 79, that's too big. That's no good.

So, I go down. So, I get 50.

I do the same thing over and over.

I try to go to the right: oh, 66, that's OK.

Try to go to the right: oh, 79, that's too big.

So I go down. Now I go to the right and, oh, 72: done. Otherwise, I'd go too far and try to go down and say, oops, element must not be there. It's a very simple search algorithm: same as here except just remove the L_1 and L_2.

Go right until that would go too far.

Then go down. Then go right until we'd go too far, and then go down. You might have to do this log n times. In each level, you're clearly only walking a couple of steps because the ratio between these two sizes is only two.

So, this will cost two log n for search.

Good, I mean, so that was to check because we were using intuition over here; a little bit shaky.

So, this is an ideal skip list, we have to support insertions and deletions. As soon as we do an insert and delete, there's no way we're going to maintain the structure.

It's a bit too special. There is only one of these where everything is perfectly spaced out, and everything is beautiful. So, we can't do that.

We're going to maintain roughly this structure as best we can.

And, if anyone of you knows someone in New York City subway planning, you can tell them this.

OK, so: skip lists. So, I mean, this is basically our data structure. You could use this as a starting point, but then you start using skip lists. And, we need to somehow implement insertions and deletions, and maintain roughly this structure well enough that the search still costs order log n time. So, let's focus on insertions.

If we do insertions right, it turns out deletions are really trivial.

And again, this is all from first principles.

We're not allowed to use anything fancy.

But, it would be nice if we used some good chalk.

This one looks better. So, suppose you want to insert an element, x. We said how to search for an element. So, how do we insert it?

Well, the first thing we should do is figure out where it goes.

So, we search for x. We call search of x to find where x fits in the bottom list, not just any list.

Pretty easy to find out where it fits in the top list.

That takes, like, constant time.

What we want to know: because the top list has constant length, we want to know where x goes in the bottom list. So, let's say we want to insert a search for 80. Well, it is a bit too big.

Let search for 75. So, we'll find the 75 fits right here between 72 and 79 using the same path.

OK, if it's there already, we complain because I'm going to assume all keys are distinct for now just so the picture stays simple. But this works fine even if you are inserting the same key over and over.

So, that seems good. One thing we should clearly do is insert x into the bottom list.

We now know where it fits. It should go there.

Because we want to maintain this invariant, that the bottom list contains all the elements.

So, there we go. We've maintained the invariant.

The bottom list contains all the elements.

So, we search for 75. We say, oh, 75 goes here, and we just sort of link in 75. You know how to do a linked list, I hope. Let me just erase that pointer.

All the work in implementing skip lists is the linked list manipulation. Is that enough?

No, it would be fine for now because now there's only a chain of length three here that you'd have to walk over if you're looking for something in this range.

But if I just keep inserting 75, and 76, than 76 plus epsilon, 76 plus two epsilon, and so on, just pack a whole bunch of elements in here, this chain will get really long. Now, suddenly, things are not so balanced. If I do a search, I'll pay an arbitrarily long amount time here to search for someone. If I insert k things, it'll take k time. I want it to stay log n.

If I only insert log n items, it's OK for now.

What I want to do is decide which of these lists contain 75.

So, clearly it goes on the bottom.

Every element goes in the bottom.

Should it go up a level? Maybe.

It depends. It's not clear yet.

If I insert a few items here, definitely some of them should go on the next level. Should I go to levels up?

Maybe, but even less likely. So, what should I do?

Yeah? Right, so you maintain the ideal partition size, which may be like the length of this chain. And you see, well, if that gets too long, then I should split it in the middle, promote that guy up to the next level, and do the same thing up here. If this chain gets too long between two consecutive next level express stops, then I'll promote the middle guy.

And that's what you'll do in your problem set.

That's too fancy for me. I don't need no stinking counters. What else could I do?

I could try to maintain the ideal skip list structure.

That will be too expensive. Like I say, 75 is the guy that gets promoted, and this guy gets demoted all the way down. But that will propagate everything to the right. And that could cost linear time for update. Other idea?

If I only want half of them to go up, I could flip a coin.

Good idea. All right, for that, I will give you a quarter. It's a good one.

It's the old line state, Maryland.

There you go. However, you have to perform some services for that quarter, namely, flip the coin.

Can you flip a coin? Good.

What did you get? Tails, OK, that's the first random bit. But we are going to do is build a skip list. Maybe I should tell you how first. OK, but the idea is flip a coin. If it's heads, so, sorry, if it's heads, we will promote it to the next level, and flip again. So, this is an answer to the question, which other lists should store x?

How many other lists should we add x to?

Well, the algorithm is, flip a coin, and if it comes out heads, then promote x.

to the next level up, and flip again.

OK, that's key because we might want this element to go arbitrarily high. But for starters, we flip a coin. It doesn't go to the next level. Well, we'd like it to go to the next level with probability one half because we want the ratio between these two sizes to be a half, or sorry, two, depending which way you take the ratio.

So, I want roughly half the elements up here.

So, I flip a coin. If it comes up heads, I go up here. This is a fair coin.

So I want it 50-50. OK, then how many should that element go up to the next level up?

Well, with 50% probability again.

So, I flip another point. If it comes up heads, I'll go up another level. And that will maintain the approximate ratio between these two guys as being two.

The expected ratio will definitely be two, and so on, all the way up. If I go up to the top and flip a coin, it comes up heads, I'll make another level.

This is the insertion algorithm: dead simple.

The fancier one you will see on your problem set.

So, let's do it.

OK, I also need someone to generate random numbers.

Who can generate random numbers?

Pseudo-random? I'll give you a quarter.

I have one here. Here you go.

That's a boring quarter. Who would like to generate random numbers? Someone volunteering someone else: that's a good way to do it.

Here you go. You get a quarter, but you're not allowed to flip it.

No randomness for you; well, OK, you can generate bits, and then compute a number. So, give me a number.

44, can answer. OK, we already flipped a coin and I got tails. Done.

That's the insertion algorithm. I'm going to make some more space actually, put it way down here.

OK, so 44 does not get promoted because we got a tails.

So, give me another number. Nine, OK, I search for nine in this list. I should mention one other thing, sorry. I need a small change.

This is just to make sure searches still work.

So, the worry is suppose I insert something bigger and then I promote it. This would look very bad for a skip list data structure because I always want to start at the top left, and now there's no top left.

So, just minor change: just let me remember that.

The minor change is that I'm going to store a special value minus infinity in every list. So, minus infinity always gets promoted all the way to the top, whatever the top happens to be now. So, initially, that way I'll always have a top left.

Sorry, I forgot to mention that.

So, initially I'll just have minus infinity.

Then I insert 44. I say, OK, 44 goes there, no promotion, done.

Now, we're going to insert nine.

Nine goes here. So, minus infinity to nine, flip your coin, heads.

Did he actually flip it? OK, good.

He flipped it before, yeah, sure.

I'm just giving you a hard time.

So, we have nine up here. We need to maintain this minus infinity just to make sure it gets promoted along with everything else. So, that looks like a nice skip list. Flip it again.

Tails, good. OK, so this looks like an ideal skip list. Isn't that great?

It works every time. OK, give me another number.

26, OK, so I search for 26. 26 goes here.

It clearly goes on the bottom list.

Here we go, 26, and then I you raised 44.

Flip. Tails, OK, another number.

50, oh, a big one. It costs me a little while to search, and I get over here.

Flip. Heads, good.

So 50 gets promoted. Flip it again.

Tails, OK, still a reasonable number.

Another number? 12, it takes a little while to get exciting here. OK, 12 goes here between nine and 26. You're giving me a hard time here. OK, flip.

Heads, OK, 12 gets promoted. I know you have to work a little bit, but we just came here to search for 12.

So, we know that nine was the last point we went down.

So, we promote 12. It gets inserted up here.

We are just inserting into this particular linked list: nothing fancy. We link the two twelves together. It still looks kind of like a linked list. Flip again.

OK, tails, another number. Jeez. It's a good test of memory.

37, what was it, 44 and 50?

And 50 was at the next level up.

I think I should just keep appending elements and have you flip coins. OK, we just inserted 37.

Tails. OK, that's getting to be a long chain. That looks a bit worse.

OK, give me another number larger than 50.

51, good answer. Thank you.

OK, flip again. And again.

Tails. Another number.

Wait, someone else should pick a number.

It's not working. What did you say?

52, good answer. Flip.

Tails, not surprising. We've gotten a lot of heads there. OK, another number.

53, thank you. Flip.

Heads, heads, OK.

Heads, heads, you didn't flip.

All right, 53, you get the idea.

If you get two consecutive heads, then the guy goes up two levels. OK, now flip for real.

Heads. Finally.

Heads we've been waiting for. If you flipped three heads in a row, you go three levels. And each time, we keep promoting minus infinity.

Look again. Heads, oh my God.

Where were they before? Flip again.

It better be tails this time. Tails, good.

OK, you get the idea. Eventually you run out of board space. Now, it's pretty rare that you go too high. What's the probability that you go higher than log n? Another easy log computation.

Each time, I have a 50% probability of going up.

One in n probability of going up log n levels because half to the power of log n is one out of n.

So, it depends on n, but I'm not going to go too high. And, intuitively, this is not so bad. So, these are skip lists.

You have the ratios right in expectation, which is a pretty weak statement. This doesn't say anything about the lengths of these change. But intuitively, it's pretty good. Let's say pretty good on average. So, I had two semi-random processes going on here. One is picking the numbers, and that, I don't want to assume anything about.

The numbers could be adversarial.

It could be sequential. It could be reverse sorted.

It could be random. I don't know.

So, it didn't matter what he said.

At least, it shouldn't matter. I mean, it matters here.

Don't worry. You're still loved.

You still get your $0.25. But what the algorithm cares about is the outcomes of these coins.

And the probability, the statement that this data structure is fast with high probability is only about the random coins. Right, it doesn't matter what the adversary chooses for numbers as long as those coins are random, and the adversary doesn't know the coins.

It doesn't know the outcomes of the coins.

So, in that case, on average, overall of the coin flips, you should be OK. But the claim is not just that it's pretty good on average. But, it's really, really good almost always. OK, with really high probability it's log n. So, for example, with probability, one minus one over n, it's order of log n, with probability one minus one over n^2 it's log n, probability one minus one over n^100, it's order log n. All those statements are true for any value of 100. So, that's where we're going.

OK, I should mention, how do you delete in a skip list? Find the element.

You delete it all the way. There's nothing fancy with delete. Because we have all these independent, random choices, all of these elements are sort of independent from each other. We don't really care.

So, delete an element, just throw it away.

The tricky part is insertion. When I insert an element, I'm just going to randomly see how high it should go.

With probability one over two to the i, it will go to height i. Good, that's my time.

I've been having too much fun here.

I've got to go a little bit faster, OK.

So here's the theorem. Let's see exactly what we are proving first. With high probability, this is a formal notion which I will define a second.

Every search in n elements skip lists costs order of log n.

So, that's the theorem. Now I need to define with high probability. So, with high probability.

And, it's a bit of a long phrase.

So, often we will, and you can abbreviate it WHP.

So, if I have a random event, and the random event here is that every search in an n element skip list costs order log n, I want to know what it means for that event E to occur with high probability.

So this is the definition. So, the statement is that for any alpha greater than or equal to one, there is a suitable choice of constants -- -- for which the event, E, occurs with this probability I keep mentioning. So, the probability at least one minus one over n to the alpha.

So, this is a bit imprecise, but it will suffice for our purposes. If you want to really formal definition, you can read the lecture notes.

There are special lecture notes for this lecture on the stellar site. And, there's the PowerPoint notes on the SMA site. But, right, there's a bit of a subtlety in the choice of constants here.

There is a choice of this constant.

And there's a choice of this constant.

And, these are related. And, there's alpha, which we get to whatever we want.

But the bottom line is, we get to choose what probability we want this to be true.

If I want it to be true, with probability one minus one over n^100, I can do that. I just sat alpha to a hundred, and up to this little constant that's going to grow much slower than n to the alpha. I get the error probability.

So this thing is called the error probability.

The probability that I fail is polynomially small, for any polynomial I want. Now, with the same data structure, right, I fixed the data structure.

It doesn't depend on alpha. Anything you want, any alpha value you want, this data structure will take order of log n time. Now, this constant will depend on alpha. So, you know, you want error probability one over n^100 is probably going to be, like, 100 log n. It's still log n.

OK, this is a very strong claim about the tale of the distribution of the running time of search, very strong.

Let me give you an idea of how strong it is.

How many people know what Boole's inequality is?

How many people know what the union bound is in probability?

You should. It's in appendix c.

Maybe you'll know it by the theorem.

It's good to know it by name. It's sort of like linearity of expectations. It's a lot easier to communicate to someone. Linearity of expectations: instead of saying, you know that thing where you sum up all the expectations of things, and that's the expectation of the sum? It's a lot easier to say linearity of expectation. So, let me quiz you in a different way. So, if I take a bunch of events, and I take their union, either this happens or this happens, or so on. So, this is the inclusive OR of k events. And, instead, I look at the sum of the probabilities of those events.

OK, easy question: are these equal?

No, unless they are independent.

But can I say anything about them, any relation?

Smaller, yeah. This is less than or equal to that. OK, this should be intuitive to you from a probability point of view.

Look at the textbook. OK: very basic result, trivial result almost. What does this tell us?

Well, suppose that E_i is some kind of error event.

We don't want it to happen. OK, and suppose, mix some letters here. Suppose I have a bunch of events which occur with high probability.

OK, call those E_i complement. So, suppose, so this is the end of that statement, E_i complement occurs with high probability. OK, so then the probability of E_i is very small, polynomially small.

One over n to the alpha for any alpha I want.

Now, suppose I take a whole bunch of these events, and let's say that k is polynomial in n.

So, I take a bunch of events, which I'd like to happen.

They all occur with high probability.

There is only polynomially many of them.

So let's say, let me give this constant a name. Let's call it c.

Let's say I take n to the c such events.

Well, what's the probability that all those events occur together? Because they should rest of the time occurred together because each one occurs most of the time, occurs with high probability.

So, I want to look at E_1 bar intersect, E_2 bar, and so on. So, each of these occurs as high probability. What's the chance that they all occur? It's also with high probability. I'm changing the alpha.

So, the union bound tells me the probability of any one of these failing, the probability of this failing, or this failing, or this failing, which is this thing, is, at most, the sum of the probabilities of each failure.

These are the error probabilities.

I know that each of them is, at most, one over n to the alpha, with a constant in front. If I add them all up, there's only n to the c of them.

So, I take this error probability, and I multiply by n to the c. So, I get like n to the c over n to the alpha, which is one over n to the alpha minus c. I can set alpha as big as I want. So, I said it much, much bigger than c, and this event occurs with high probability. I sort of made a mess here, but this event occurs with high probability because of this.

Whatever the constant is here, however many events I'm taking, I just set alpha to be bigger than that.

And, this event will occur with high probability, too. So, when I say here that every search of cost order log n with high probability, not only do I mean that if you look at one search, it costs order log n with high probability.

You look at another search, and it costs log n with high probability. I mean, if you take every search, all of them take order log n time with high probability. So, this event that every single search you do takes order log n, is true with high probability estimate the number of searches you are doing is polynomial in n. So, I'm assuming that I'm not using this data structure forever, just for a polynomial amount of time. But, who's got more than a polynomial amount of time anyway?

This is MIT. So, hopefully that's clear.

We'll see it a few more times. Yeah?

The algorithm doesn't depend on Alpha.

The question is how do you choose alpha in the algorithm.

So, we don't need to. This is just sort of for an analysis tool. This is saying that the farther out you get, so you say, well, what's the probability that more than ten log n. Well, it's like one over n^10.

Let's say it's linear. Well, what's the chance that you're more than 20 log n? Well that's one over n^20.

So, the point is the tail of this distribution is getting a really small, really fast.

And, such using alpha is more like sort of for your own feeling good. OK, you can set it to 100, and then n is at least two. So, that's like one over 2^100 chance that you fail. That's damn small.

If you've got a real random number generator, the chance that you're going to hit one over 2^200 is pretty tiny, right? So, let's say you set alpha to 256, which is always a good number.

2^256 is much bigger than the number of particles in the known universe, so, the light matter.

So, actually I think this even accounts for some notion of dark matter. So, this is really, really, really big. So, the chance that you pick a random particle in the universe that happens to be your favorite particle, this one right here, that's over one over 2^256, or even smaller. So, set alpha to 256, the chance to your algorithm takes more than order log n time is a lot smaller than the chance that a meteor strikes your computer at the same time that it has a flooding point error, at the same time that the earth explodes because they're putting a transport through this part of the solar system at the same time, I mean, I could go on, right? It's really, really unlikely that you are more than log n.

And how unlikely: you get to choose.

But it's just in the analysis the algorithm doesn't depend on it. It's the same algorithm, very cool. Sometimes, with high probability, bounds depends on alpha.

I mean, the algorithm depends on alpha.

But here, it will not. OK, away we go.

So now you all understand the claim.

So let's do a warm up. We will also need this fact.

But it's pretty easy. The lemma is that with high probability, the number of levels in the skip list is order log n. I think it's order log n, certainly. So, how do we prove that something happens with high probably?

Compute the probability that it happened; show that it's high.

Even if you don't know what high probability means, in fact, I used to ask that earlier on.

So, let's compute the chance that it doesn't happen, the error probability, because that's just a one minus the cleaner. So, I'd like to say, let's say, that it's, at most, c log n levels.

So, what's the error probability for that event?

This is sort of an event. I'll put it in squiggles just for, all set. This is the probability that they are strictly greater than c log n levels.

So, I want to say that that probability is particularly small, polynomially small. Well, how do I make levels?

When I insert an element, the probability half, it goes up. And, the number of levels in the skip list is the max over all the elements of how high it goes up. But, max, oh, that's a mess. All right, you can compute the expectation of the max if you have a bunch of unknown variables; there is expectation there is a constant, and you take the max. It's like log in and expectation, but we want a much stronger statement.

And, we have this Boole's inequality that says I have a bunch of things, polynomially many things.

Let's say we have n items. Each one independently, I don't even care if it's a dependent.

If it goes up more than c log n, yeah, the number of levels is more than c log n. So, this is, at most, and then I want to know, do any of those events happen for any of the n elements?

So, I just multiplied by n. It's certainly, at most, n times the probability that x gets promoted, this much here, greater than or equal to log n times. OK, if I pick, for any element, x, because it's the same for each element. They are done independently.

So, I'm just summing over x here, and that's just a factor of n. Clear?

This is Boole's inequality. Now, what's the probability that x gets promoted c log n times?

We did this before for log n. It was one over n.

For c log n, it's one over n to the c.

OK, this is n times two. Let's be nicer: one half to the power of c log n.

One half to the power of c log n is one over two to the c log n. The log n comes out here, becomes an n. We get n to the c.

So, this is n divided by n to the c, which is n to the c minus one. And, I get to choose c to be whatever I want. So, I choose c minus one to be alpha. I think exactly that.

Oh, sorry, one over n to the c minus one.

Thank you. It better be small.

This is an upper bound. So, probability is polynomially small. I get to choose, and this is a bit of the trik. I'm choosing this constant to be large, large enough for alpha.

The point is, as c grows, alpha grows.

Therefore, I can set alpha to be whatever I want, set c accordingly. So, there's a little bit more words that have to go here. But, they're in the notes.

I can set alpha to be as large as I want.

So, I can make this probability as small as I want in the polynomial sets. So, that's it.

Number of levels, order log n: wasn't that easy? Rules and equality, the point is that when you're dealing with high probability, use Boole's inequality. And, anything that's true for one element is true for all of them, just like that.

Just lose a factor of n, but that's just one in the alpha, and alpha is big: big constant, but it's big. OK, so let's prove the theorem.

High probability searches cost order log n.

We now know the height is order log n, but it depends how balanced this thing is. It depends how long the chains are to really know that a search costs log n.

Just knowing a bound on the height is not enough, unlike a binary tree. So, we have one cool idea for this analysis. And it's called backwards analysis. So, normally you think of a search as starting in the top left corner going left and down until you get to the item that you're looking for.

I'm going to look at the reverse process.

You start at the item you're looking for, and you go left and up until you get to the top left corner.

The number of steps in those two walks is the same.

And, I'm not implementing an algorithm here, I'm just doing analysis. So, those are the same processes, just in reverse. So, here's what it looks like.

You have a search, and it starts, which really means that it ends at a node in the bottom list.

Then, each time you visit a node in this search, you either go left or up. And, when do you go left or up?

Well, it depends with the coin flip was.

So, if the node wasn't promoted at this level.

So, if it wasn't promoted higher, and that happened exactly when we got a tails. Then, we go left, which really means we came from the left.

Or, if we got a heads, so if this node was promoted to the next level, which happened whenever we got a heads at that particular moment.

This is in the past some time when we did the insertion.

Then we go, or came from, up.

And, we stop at the root. This is really where we start; same thing. So, either at the root or I'm also going to think of this as stopping at minus infinity.

OK, that was a bit messy, but let me review.

So, normally we start up here. Well, just looking at everything backwards, and in brackets is what's really happening. So, this search ends at the node you were looking for. It's always in the bottom list.

Then it says, well, was this node promoted higher? If it was, I came from above.

If not, I came to the left. It must have been in the bottom chain somewhere. OK, and that's true at every node you visit. It depends whether that quite slipped heads or tails at the time that you inserted that node into that level. But, these are just a bunch of events. I'm just going to check, what is the probability that its heads, and what is the probability that a tails? It's always a half.

Every time I look at a coin flip, when it was flipped, there was a probability of half going out of their way.

That's the magic. And, I'm not using that these events are independent anyway. For every element that I search for, for every value, x, that's another search.

Those events may not be independent.

I can still use Boole's inequality and conclude that all of them are order log n with high probability.

As long as I can prove that any one event happens with high probability. So, I don't need independence between, I knew that these coin flips in a single search are independent, but everything else, for different searches I don't care. So, how long can this process go on? We want to know how many times can I make this walk? Well, when I hit the root node, I'm done. Well, how quickly would I hit the root node? Well, with probability, a half, I go up each step. The number of times I go up is, at most, the number of levels minus one.

And that's order log n with high probability.

So, this is the only other idea.

So, we are now improving this theorem.

So, the number of up moves in a search, which are really down moves, but same thing, is less than the number of levels. Certainly, you can't go up more than there are levels in the search.

And in insert, you can go arbitrarily high.

But a search: as high as you can go.

And this is, at most, c log n with high probability. This is what we proved in the lemma. So, we have a bound on the number of up moves. Half of the moves, roughly, are going to be up moves.

So, this pretty much down to the number of moves.

Not quite. So, what this means is that with high probability, so this is the same probability, but I could choose that as high as I want by setting c large enough. The number of moves, in other words, the cost of the search is at most the number of coin flips until we get c long n heads, right, because in every step of the search, I make a move, and then I flip another coin, conceptually.

There is another independent coin lying there.

And it's either heads or tails. Each of those is independent.

So, how many independent coin flips does it take until I get c log n heads? The claim is that that's order log n with high probability. But we need to prove that.

So, this is a claim. So, if you just sit there with a coin, and you want to know how many times does it take until I get c log n heads, the claim is that that number is order log n with high probability.

As long as I prove that, I know that the total number of steps I make, which is the number of heads and tails is order log n because I definitely know the number of heads is, at most, c log n.

The claim is that the number of tails can't be too much bigger.

Notice, I can't just say c here.

OK, it's really important that I have log n.

Why? Because with high probability, it depends on n. This notion depends on n.

Log n: it's true. Anything bigger that log n: it's true, like n. If I put n here, this is also true. But, if I put a constant or a log log n, this is not true. It's really important that I have log n here because my notion of high probability depends on what's written here. OK, it's clear so far.

We're almost done, which is good because I just ran out of time. Sorry, we're going to go a couple minutes over. So, I want to compute the error probability here. So, I want to compute the probability that there is less than c log n heads.

Let me skip this step. So, I will be approximate and say, what's the probability that there is, at most, c log n heads? So, I need to say how many coins we are flipping here for what this event is.

So, I need to specify this constant.

Let's say we flip ten c log n coins.

Now I want to look at the error probability under that event.

The probability that there is at most c log n heads among those ten c log n flips. So, the claim is this should be pretty small. It's going to depend on ten.

Then I'll choose ten to be arbitrarily large, and I'll be done, OK, make my life a little bit easier. Well, I would ask you normally, but this is 6.042 material. So, what's the probability that we have, at most, this many heads?

Well, that means that nine c log n of the coins, because there are ten c log n flips, c log n heads at most, nine c log n at least better be tails.

So this is the probability that all those other guys become tails, which is already pretty small.

And then, there is this permutation thing.

So, if I had exactly c log n heads, this would be the number of ways to rearrange c log n heads among ten c log n coin flips. OK, that's just the number of permutations. So, this is a bit big, which is kind of annoying. This is really, really small. The claim is this is much smaller than that is big.

So, this is just some math. I'm going to whiz through it.

So, you don't have to stay too long.

But you should go over it. You should know that y choose x is, at most, ey over x to the x, good fact.

Therefore, this is, at most, ten c log n over c log n, also known as ten. These cancel.

There's an e out here. And then I raise that to the c log n power. OK, then I divide by two to the power, nine c log n. OK, so what's this?

This is e times ten to the c log n divided by two to the nine c log n. OK, claim this is very big.

This is not so big, because I have a nine here.

So, let's work it out. This e times ten, that's a good number, we can put upstairs.

So, we get log of e times ten, ten times, e, and then c log n. And then, we have over two to the nine c log n. So, we have this two to the c log n in both cases. So, this is two to the log, ten e minus nine, c, log n: some basic algebra.

So, I'm going to set, not quite.

This is one over two to the nine minus log: so, just inverting everything here, negating the sign in here.

And, this is my alpha because the rest is n.

So, this is one over n to the alpha when alpha is this particular value: nine minus log of ten times e times c. It's a bit of a strange thing.

But, the point is, as ten goes to infinity, nine here is the number one smaller than ten, right? We subtracted one somewhere along the way. So, as ten goes to infinity, this is basically, this is ten minus one.

This is log of ten times e. e doesn't really matter.

The point is, this is logarithmic in ten.

This is linear in ten. The thing that's linear in ten is much bigger than the thing that's logarithmic in ten.

This is called abusive notation.

OK, as ten goes to infinity, this goes to infinity, gets bigger. And, there is a c out here.

But, for any value of c that you want, whatever value of c you wanted in that claim, I can make alpha arbitrarily large by changing the constant in the big O, which here was ten. OK, so that claim is true with high probability. Whatever probability you want, which tells you alpha, you set a constant effort of the log N to be this number, which grows, and you're done. You get the claim that is order log N heads, order log N flips with the high probability, therefore. [None of the steps?] in the search is order log N with high probability.

Really cool stuff; read the notes.

Sorry I went so fast at the end.

Free Downloads

Video

• iTunes U (MP4 - 179MB)
• Internet Archive (MP4 - 352MB)

Free Streaming

• VideoLectures.net

Subtitle

• English - US (SRT)