WEBVTT

00:00:00.080 --> 00:00:02.430
The following content is
provided under a Creative

00:00:02.430 --> 00:00:03.810
Commons license.

00:00:03.810 --> 00:00:06.060
Your support will help
MIT OpenCourseWare

00:00:06.060 --> 00:00:10.150
continue to offer high quality
educational resources for free.

00:00:10.150 --> 00:00:12.700
To make a donation or to
view additional materials

00:00:12.700 --> 00:00:16.600
from hundreds of MIT courses,
visit MIT OpenCourseWare

00:00:16.600 --> 00:00:17.305
at ocw.mit.edu.

00:00:25.941 --> 00:00:27.316
PROFESSOR: Everyone,
welcome back

00:00:27.316 --> 00:00:28.565
to Poker Theory and Analytics.

00:00:28.565 --> 00:00:31.760
We're very lucky today to have a
guest speaker, Matt Hawrilenko.

00:00:31.760 --> 00:00:34.340
Matt's background is
really interesting.

00:00:34.340 --> 00:00:37.130
He was a Princeton grad who
went to work for a Cisco

00:00:37.130 --> 00:00:38.880
International group
for a couple of years,

00:00:38.880 --> 00:00:40.838
and then he left to become
a professional poker

00:00:40.838 --> 00:00:41.640
player full time.

00:00:41.640 --> 00:00:44.180
He was considered one of the
best limit hold'em players

00:00:44.180 --> 00:00:47.165
in the world in addition to
winning a World Series of Poker

00:00:47.165 --> 00:00:48.070
bracelet.

00:00:48.070 --> 00:00:49.720
More recently he
retired from poker

00:00:49.720 --> 00:00:52.380
and is focusing on
clinical psychology

00:00:52.380 --> 00:00:54.910
where he studies in
Boston at both Clark

00:00:54.910 --> 00:00:56.730
University and Harvard.

00:00:56.730 --> 00:00:59.570
So with that, I'll pass it
along to Matt Hawrilenko.

00:00:59.570 --> 00:01:01.920
[APPLAUSE]

00:01:03.906 --> 00:01:04.739
MATT HAWRILENKO: OK.

00:01:04.739 --> 00:01:05.238
Hey.

00:01:05.238 --> 00:01:07.230
So I'm super excited
to be here today.

00:01:07.230 --> 00:01:10.280
And as I understand, I've been
told that I'm your second game

00:01:10.280 --> 00:01:12.680
theory guy.

00:01:12.680 --> 00:01:15.391
And I guess Bill, Bill
Chen, one of my best friends

00:01:15.391 --> 00:01:16.890
in poker who I've
done a lot of work

00:01:16.890 --> 00:01:18.807
with came in and talked
to you guys last week.

00:01:18.807 --> 00:01:20.640
I'm going to take a
slightly different angle

00:01:20.640 --> 00:01:21.930
than Bill did last week.

00:01:21.930 --> 00:01:23.880
So first I kind of
just want to get

00:01:23.880 --> 00:01:25.290
a sense of who's in the room.

00:01:25.290 --> 00:01:27.530
So my presumption
is that there might

00:01:27.530 --> 00:01:30.780
be sort of like widely
varying levels of experience

00:01:30.780 --> 00:01:32.080
with game theory.

00:01:32.080 --> 00:01:37.450
So if you just don't mind
kind of helping me out,

00:01:37.450 --> 00:01:39.890
so who here has seen a
game theory game before,

00:01:39.890 --> 00:01:41.950
like prisoner's dilemma,
something like that?

00:01:41.950 --> 00:01:42.450
OK.

00:01:42.450 --> 00:01:44.702
So most of us.

00:01:44.702 --> 00:01:47.160
Who here has actually solved
a couple of game theory games,

00:01:47.160 --> 00:01:50.376
like with pen and paper,
matrix algebra, whatever?

00:01:50.376 --> 00:01:51.252
OK.

00:01:51.252 --> 00:01:52.900
About half.

00:01:52.900 --> 00:01:55.680
Has anyone actually taken
a game theory class?

00:01:55.680 --> 00:01:56.320
All right.

00:01:56.320 --> 00:01:57.180
Great.

00:01:57.180 --> 00:02:00.410
So I'm kind of
hoping this talk is

00:02:00.410 --> 00:02:04.620
sort of going to be equally
useful for everyone,

00:02:04.620 --> 00:02:06.730
but we'll find out.

00:02:06.730 --> 00:02:09.850
So Bill talked to you guys
about Cepheus last week.

00:02:09.850 --> 00:02:13.270
I'm going to talk about
something a little different

00:02:13.270 --> 00:02:14.520
than Bill usually talks about.

00:02:14.520 --> 00:02:17.700
So I'm going to try to talk
about how to play good poker.

00:02:17.700 --> 00:02:20.820
And so if we think of this
problem of playing good poker

00:02:20.820 --> 00:02:22.100
is this big game space.

00:02:22.100 --> 00:02:26.700
There are two ways that
people generally approach it

00:02:26.700 --> 00:02:29.430
in that they'll take a
read-based approach, which

00:02:29.430 --> 00:02:31.990
is pretty much what
everybody does, right?

00:02:31.990 --> 00:02:34.230
You try to figure out
what the other guy has,

00:02:34.230 --> 00:02:37.185
what he might have,
and go from there.

00:02:37.185 --> 00:02:38.890
You can take a game
theoretic approach.

00:02:38.890 --> 00:02:40.850
So I'm going to try
today discriminate

00:02:40.850 --> 00:02:43.080
a little bit between
these two approaches,

00:02:43.080 --> 00:02:45.460
and I'm going to talk about
why, whatever level of poker

00:02:45.460 --> 00:02:48.250
you're at, game theory is
something they can complement

00:02:48.250 --> 00:02:50.190
your game or just be your game.

00:02:50.190 --> 00:02:51.480
For me it is my game.

00:02:54.555 --> 00:02:56.180
And broadly speaking,
the reason that I

00:02:56.180 --> 00:02:59.570
think game theory is
particularly useful for poker

00:02:59.570 --> 00:03:01.290
goes something like this.

00:03:01.290 --> 00:03:05.180
So one of the things that
I love to do off the table,

00:03:05.180 --> 00:03:05.722
I do jujitsu.

00:03:05.722 --> 00:03:07.096
Which if you guys
don't know what

00:03:07.096 --> 00:03:09.510
that is, it's kind of like
wrestling where you're trying

00:03:09.510 --> 00:03:12.210
to choke the guy or hyperextend
their arm or something

00:03:12.210 --> 00:03:13.190
like that.

00:03:13.190 --> 00:03:15.900
And when you start
out doing jujitsu,

00:03:15.900 --> 00:03:18.500
you start out as a white belt,
and you're competing against,

00:03:18.500 --> 00:03:20.770
rolling with other white belts.

00:03:20.770 --> 00:03:23.620
And you can learn this
whole repertoire of moves.

00:03:23.620 --> 00:03:26.410
And tons of stuff works
against white belts, right?

00:03:26.410 --> 00:03:27.910
Like, whatever you
do, it's probably

00:03:27.910 --> 00:03:29.900
going to work
against white belts.

00:03:29.900 --> 00:03:32.960
As you get better
or as you start

00:03:32.960 --> 00:03:35.310
going against better
competition-- blue belts,

00:03:35.310 --> 00:03:37.440
purple, brown,
black belts-- a lot

00:03:37.440 --> 00:03:41.370
of the moves that worked against
white belts don't work anymore.

00:03:41.370 --> 00:03:44.200
Not only do they
not work, they can

00:03:44.200 --> 00:03:47.630
tend to get you in
quite a lot of trouble.

00:03:47.630 --> 00:03:49.370
So I think that the
best way to approach

00:03:49.370 --> 00:03:51.740
this game is from
day one, you're not

00:03:51.740 --> 00:03:53.290
training to be white belts.

00:03:53.290 --> 00:03:56.080
You're training to be
black belts, right?

00:03:56.080 --> 00:03:59.070
You want to learn the moves
that work all the way along

00:03:59.070 --> 00:04:00.527
against the best competition.

00:04:00.527 --> 00:04:02.860
And I think it's sort of the
same way with poker, right?

00:04:02.860 --> 00:04:05.026
So if you think about it,
if you play in a home game

00:04:05.026 --> 00:04:06.810
or whatever with a
bunch of buddies,

00:04:06.810 --> 00:04:10.600
you might have one strategy
that works fairly well, right?

00:04:10.600 --> 00:04:14.701
Maybe you play really tight, you
bet a lot with your best hands,

00:04:14.701 --> 00:04:15.950
and you don't bluff that much.

00:04:15.950 --> 00:04:17.870
And this works really
well because most people

00:04:17.870 --> 00:04:21.450
don't really have a
sense of what's what,

00:04:21.450 --> 00:04:22.760
of the strength of hands.

00:04:22.760 --> 00:04:24.230
They overvalue their hands.

00:04:24.230 --> 00:04:28.650
So you take that
strategy to the Bellagio

00:04:28.650 --> 00:04:32.900
and you play in even mid limit
games with reasonable players,

00:04:32.900 --> 00:04:35.480
and all of a sudden you're
starting to get eaten alive.

00:04:35.480 --> 00:04:39.810
So I'm going to talk about how
game theory can sort of help

00:04:39.810 --> 00:04:42.740
you avoid this and help you
be strong and grow stronger

00:04:42.740 --> 00:04:44.199
all the way along.

00:04:44.199 --> 00:04:45.740
So now if we think
about game theory,

00:04:45.740 --> 00:04:48.720
really there are two audiences
that we could be speaking to,

00:04:48.720 --> 00:04:49.220
right?

00:04:49.220 --> 00:04:52.240
So Bill came in last
week and he talked

00:04:52.240 --> 00:04:58.740
about Cepheus, this algorithm
that solved my best game.

00:04:58.740 --> 00:05:01.100
And, well, that is sad for me.

00:05:01.100 --> 00:05:04.320
I'm not going to spend a lot
of time dwelling on that here.

00:05:04.320 --> 00:05:07.130
Rather I'm going to talk
about what humans can do,

00:05:07.130 --> 00:05:11.270
how humans can apply game theory
at the table to their game.

00:05:14.420 --> 00:05:15.950
And this is for a
couple of reasons,

00:05:15.950 --> 00:05:20.590
but maybe the bigger one is
I've had a fair bit of math,

00:05:20.590 --> 00:05:23.740
but I can't solve the kinds of
problems that Bill can solve.

00:05:23.740 --> 00:05:26.850
But my work has sort of
been on taking game theory,

00:05:26.850 --> 00:05:29.150
taking these concepts,
and really using them

00:05:29.150 --> 00:05:30.170
in a practical sense.

00:05:30.170 --> 00:05:31.760
So that's what I want
to talk about today.

00:05:31.760 --> 00:05:32.730
So if we're thinking
about this there

00:05:32.730 --> 00:05:34.320
are, again, a couple
angles we could take.

00:05:34.320 --> 00:05:36.486
We could talk about it from
a theoretic perspective.

00:05:36.486 --> 00:05:37.770
We could solve a lot of games.

00:05:37.770 --> 00:05:40.350
We could solve
dynamical systems,

00:05:40.350 --> 00:05:41.580
and see how that works.

00:05:41.580 --> 00:05:43.290
Or we could talk
about it in practice.

00:05:43.290 --> 00:05:44.539
What does game theory tell me?

00:05:44.539 --> 00:05:46.957
How does it tell me I should
think about this situation?

00:05:46.957 --> 00:05:48.540
Of course, I think
the theory practice

00:05:48.540 --> 00:05:50.370
dichotomy is a false economy.

00:05:50.370 --> 00:05:54.120
So what I want to do is I want
to spend a little bit a time

00:05:54.120 --> 00:05:56.900
solving some really, really
simple toy games, two toy

00:05:56.900 --> 00:05:57.720
games.

00:05:57.720 --> 00:06:00.070
And I want to sort of use
that to bridge the theory

00:06:00.070 --> 00:06:01.330
and practice gap.

00:06:01.330 --> 00:06:04.580
So I want to take a
couple of toy games

00:06:04.580 --> 00:06:08.180
and then apply them to a real
hand of high stakes poker

00:06:08.180 --> 00:06:08.680
I played.

00:06:08.680 --> 00:06:12.375
So I'm going to use that
hand to motivate how we might

00:06:12.375 --> 00:06:14.760
apply some of these ideas.

00:06:14.760 --> 00:06:17.780
So then finally at
the end of the day,

00:06:17.780 --> 00:06:19.350
we can come up
with a list, right?

00:06:19.350 --> 00:06:21.320
We can think of a
rule-based strategy.

00:06:21.320 --> 00:06:24.670
That's something that learning
from Cepheus might give you.

00:06:24.670 --> 00:06:29.572
It might tell you what to do
in every specific situation.

00:06:29.572 --> 00:06:30.530
But that's really hard.

00:06:30.530 --> 00:06:31.350
Poker's really big.

00:06:31.350 --> 00:06:33.390
You can't really
remember all of that.

00:06:33.390 --> 00:06:37.540
So the thing that I really hope
you guys take away from my talk

00:06:37.540 --> 00:06:41.190
today is some training
principles, some ways

00:06:41.190 --> 00:06:44.170
to think about the game which,
as you finish this class

00:06:44.170 --> 00:06:47.810
I guess in a couple of days and
move on throughout your poker

00:06:47.810 --> 00:06:51.480
careers, whatever
those might be,

00:06:51.480 --> 00:06:53.980
some tools that can
sort of help you

00:06:53.980 --> 00:06:56.880
continue working on your game.

00:06:56.880 --> 00:07:01.025
So let's just get going.

00:07:01.025 --> 00:07:05.300
So sometimes in poker we get
put in really tough spots,

00:07:05.300 --> 00:07:07.340
and it can start to
feel like we're trying

00:07:07.340 --> 00:07:09.770
to guess our way out of them.

00:07:09.770 --> 00:07:11.840
Varying degrees of guess
our way out of them.

00:07:11.840 --> 00:07:14.960
Sometimes we might be sort
of making educated guesses,

00:07:14.960 --> 00:07:16.740
sometimes they might
be less educated.

00:07:16.740 --> 00:07:19.220
But what do we do?

00:07:19.220 --> 00:07:22.450
What do you do in that really
tough spot where you just

00:07:22.450 --> 00:07:24.800
don't know what's going on?

00:07:24.800 --> 00:07:28.480
So this happened to me a
couple of years ago in-- this

00:07:28.480 --> 00:07:29.830
happens to me all the time.

00:07:29.830 --> 00:07:34.439
But this happened to me a couple
of years ago in a World Series

00:07:34.439 --> 00:07:35.480
tournament that I played.

00:07:35.480 --> 00:07:36.021
We were deep.

00:07:36.021 --> 00:07:37.229
We were in the money.

00:07:37.229 --> 00:07:39.270
And the one thing I want
you guys to keep in mind

00:07:39.270 --> 00:07:41.353
here is I'm playing this
hand against a player who

00:07:41.353 --> 00:07:43.970
is just much better than me.

00:07:43.970 --> 00:07:47.090
So that sucks.

00:07:47.090 --> 00:07:48.720
On the other hand, I have aces.

00:07:48.720 --> 00:07:49.830
So what I'm going to do is
I'm going to take you really

00:07:49.830 --> 00:07:52.790
briefly through the hand and
let you know how I was thinking

00:07:52.790 --> 00:07:55.920
about it, and we'll
kind of look at it

00:07:55.920 --> 00:07:57.550
from a couple perspectives.

00:07:57.550 --> 00:08:01.810
So we're both
really, really deep.

00:08:01.810 --> 00:08:04.666
The blinds are 12 and
24,000 with a $3,000 ante,

00:08:04.666 --> 00:08:05.540
and we're six handed.

00:08:05.540 --> 00:08:07.206
So I raise two off
the button with aces.

00:08:07.206 --> 00:08:09.070
I raise a little bit, he calls.

00:08:09.070 --> 00:08:10.640
Flop comes, king, jack, eight.

00:08:10.640 --> 00:08:13.190
He checks, I bet, he calls.

00:08:13.190 --> 00:08:13.940
It turns a five.

00:08:13.940 --> 00:08:15.880
He checks, I bet
about 2/3 of the pot.

00:08:15.880 --> 00:08:17.800
He calls.

00:08:17.800 --> 00:08:19.300
And then the river's a king.

00:08:19.300 --> 00:08:21.470
And then he reaches for
his chips, and I feel good.

00:08:21.470 --> 00:08:23.570
I'm like, oh, money.

00:08:23.570 --> 00:08:29.070
And then he bets just about
$1.1 million to a $700,000 pot.

00:08:29.070 --> 00:08:32.095
And now I'm like, I don't know.

00:08:32.095 --> 00:08:33.086
I have no idea.

00:08:33.086 --> 00:08:34.169
So he's betting I'm happy.

00:08:34.169 --> 00:08:35.419
I see the amount.

00:08:35.419 --> 00:08:37.340
I'm confused.

00:08:37.340 --> 00:08:40.530
So here's how the hand looks.

00:08:40.530 --> 00:08:43.882
So I'm sitting
there and I'm just

00:08:43.882 --> 00:08:45.340
trying to get a
sense of his range,

00:08:45.340 --> 00:08:48.210
and I'm thinking, like,
bet of 1.5 times the pot.

00:08:48.210 --> 00:08:49.590
What does that mean?

00:08:49.590 --> 00:08:51.570
What kind of hands is
he betting so much with?

00:08:51.570 --> 00:08:53.540
Why isn't he putting me all in?

00:08:53.540 --> 00:08:54.950
Why is it a little below all in?

00:08:54.950 --> 00:08:57.350
Is he trying to entice me?

00:08:57.350 --> 00:08:59.930
Or maybe he's trying to
save $300,000 if I fold.

00:08:59.930 --> 00:09:03.100
All these thoughts are
going through my head.

00:09:03.100 --> 00:09:06.045
So before we talk about what
I did, I'm actually curious.

00:09:06.045 --> 00:09:08.170
Are you guys happy with
the information I gave you?

00:09:08.170 --> 00:09:10.769
Is there more information
that you want?

00:09:10.769 --> 00:09:11.810
What do you want to know?

00:09:11.810 --> 00:09:14.445
AUDIENCE: How big is the
gap between getting out

00:09:14.445 --> 00:09:15.831
now and getting out next?

00:09:15.831 --> 00:09:17.330
MATT HAWRILENKO:
Ooh, good question.

00:09:17.330 --> 00:09:21.519
So let's just say it's
pretty linear at this point.

00:09:21.519 --> 00:09:22.060
AUDIENCE: OK.

00:09:24.935 --> 00:09:27.132
MATT HAWRILENKO: Anything
else you want to know?

00:09:27.132 --> 00:09:28.310
Oh, you want to know lots.

00:09:28.310 --> 00:09:28.920
What do you want to know?

00:09:28.920 --> 00:09:30.870
AUDIENCE: Has he
shown down big bluffs?

00:09:30.870 --> 00:09:33.010
MATT HAWRILENKO: Has he
shown down big bluffs?

00:09:33.010 --> 00:09:37.342
Good question, we've played
with him a little bit,

00:09:37.342 --> 00:09:39.950
and I don't actually recall.

00:09:39.950 --> 00:09:43.335
But he's certainly capable
of showing down big bluffs.

00:09:43.335 --> 00:09:45.800
I fancied myself a pretty
good player at the time,

00:09:45.800 --> 00:09:48.799
and he is someone that
I would consider one

00:09:48.799 --> 00:09:50.090
of the top players in the game.

00:09:53.400 --> 00:09:54.700
Anything else?

00:09:54.700 --> 00:09:55.250
Anyone else?

00:09:55.250 --> 00:09:55.833
Any questions?

00:09:58.770 --> 00:09:59.270
Right.

00:09:59.270 --> 00:10:00.350
And so it sounds
like what you're

00:10:00.350 --> 00:10:02.350
trying to do is you're
trying to get a sense of,

00:10:02.350 --> 00:10:03.401
what is his range?

00:10:03.401 --> 00:10:05.150
Does he have a lot of
bluffs in this spot?

00:10:05.150 --> 00:10:08.700
Does he have any
bluffs in this spot?

00:10:08.700 --> 00:10:11.587
And I think that's how
pretty much most of us

00:10:11.587 --> 00:10:12.670
tend to approach the game.

00:10:12.670 --> 00:10:15.580
It's most natural way
to approach the game.

00:10:15.580 --> 00:10:16.810
Is he bluffing, isn't he?

00:10:16.810 --> 00:10:20.030
Does he have too
many bluffs here?

00:10:20.030 --> 00:10:21.530
Out of curiosity,
does anyone want

00:10:21.530 --> 00:10:24.129
to stare him down for tells?

00:10:24.129 --> 00:10:25.420
Nobody wants to stare him down.

00:10:25.420 --> 00:10:26.795
You might want to
stare him down.

00:10:26.795 --> 00:10:28.238
This guy wants to
stare him down.

00:10:28.238 --> 00:10:30.437
[LAUGHTER]

00:10:30.437 --> 00:10:32.270
This guy definitely
wants to stare him down.

00:10:35.390 --> 00:10:38.237
This guy wants to stare
him down for sure.

00:10:38.237 --> 00:10:40.070
By the way, if you get
this look, good work.

00:10:40.070 --> 00:10:41.320
It means you've done your job.

00:10:41.320 --> 00:10:42.210
It's uncomfortable.

00:10:42.210 --> 00:10:45.560
But I want to tell you exactly
what he's thinking right now.

00:10:45.560 --> 00:10:47.850
What he's thinking goes
something kind of like this.

00:10:47.850 --> 00:10:50.290
[MUSIC PLAYING]

00:10:53.086 --> 00:10:54.460
But it can feel
like this, right?

00:10:54.460 --> 00:10:58.660
Tough players can put us in
spots where we're just thinking

00:10:58.660 --> 00:11:00.870
about monkeys clashing symbols.

00:11:00.870 --> 00:11:02.400
And when we don't
have a repertoire,

00:11:02.400 --> 00:11:05.700
I think, that we start
reading really heavily

00:11:05.700 --> 00:11:07.970
into small signals, right?

00:11:07.970 --> 00:11:10.320
We're trying to
figure something out.

00:11:10.320 --> 00:11:12.730
So generally speaking,
my view, there

00:11:12.730 --> 00:11:14.440
are some tells that are OK.

00:11:14.440 --> 00:11:19.480
There are some tells that kind
of work against weak players

00:11:19.480 --> 00:11:22.190
and less and less
against better players.

00:11:22.190 --> 00:11:26.810
But these are small signals
in a whole lot of noise.

00:11:26.810 --> 00:11:29.570
Very small signal in a
very noisy environment.

00:11:29.570 --> 00:11:33.840
Certainly not something to
build a career on, right?

00:11:33.840 --> 00:11:36.365
In the big games,
you rarely see tells.

00:11:36.365 --> 00:11:38.740
You just don't see them enough
for that to be profitable.

00:11:41.370 --> 00:11:43.650
But actually some people
do build careers on them.

00:11:43.650 --> 00:11:46.720
So FBI interrogators
build careers

00:11:46.720 --> 00:11:48.640
on these kinds of tells.

00:11:48.640 --> 00:11:50.444
So how do they do?

00:11:50.444 --> 00:11:52.360
Well, we actually have
some data in that spot.

00:11:52.360 --> 00:11:54.068
There have been a
whole series of studies

00:11:54.068 --> 00:11:56.670
where basically the paradigm
is you bring in some FBI

00:11:56.670 --> 00:11:59.150
interrogators, and then you
bring in some random people

00:11:59.150 --> 00:12:00.190
from the street.

00:12:00.190 --> 00:12:03.020
And you have them watch
someone interview a person.

00:12:03.020 --> 00:12:05.270
And then at the end of the
day you have to figure out,

00:12:05.270 --> 00:12:09.940
is this person telling the
truth, or is this person lying?

00:12:09.940 --> 00:12:11.430
How do you think they do?

00:12:11.430 --> 00:12:12.140
AUDIENCE: Same.

00:12:12.140 --> 00:12:13.056
MATT HAWRILENKO: Same?

00:12:13.056 --> 00:12:13.930
Yeah.

00:12:13.930 --> 00:12:15.500
Same in every category.

00:12:15.500 --> 00:12:18.820
So everyone is basically
chance, or maybe like 53%.

00:12:18.820 --> 00:12:20.940
So they do exactly the
same as the random people

00:12:20.940 --> 00:12:23.940
off the street, these
body language experts,

00:12:23.940 --> 00:12:25.960
except for one difference.

00:12:25.960 --> 00:12:29.960
The one difference is they
are way more confident

00:12:29.960 --> 00:12:31.521
that they are right, OK?

00:12:31.521 --> 00:12:33.270
So you ask them their
level of confidence,

00:12:33.270 --> 00:12:35.200
and most people are
like, eh, I don't know.

00:12:35.200 --> 00:12:36.400
Like 50%.

00:12:36.400 --> 00:12:40.110
The interrogators are, like,
90% sure that they're right.

00:12:40.110 --> 00:12:43.575
So they're not alone, right?

00:12:43.575 --> 00:12:46.200
So I don't know if you guys have
heard of self-assessment bias.

00:12:46.200 --> 00:12:48.210
It's one of my favorite biases.

00:12:48.210 --> 00:12:50.370
So one more study.

00:12:50.370 --> 00:12:54.620
So it was sort of motivated by
this old study of GE engineers.

00:12:54.620 --> 00:12:56.200
It's like 30 years ago.

00:12:56.200 --> 00:13:02.380
And what they do is they ask,
OK, of all the engineers at GE,

00:13:02.380 --> 00:13:03.300
where do you rank?

00:13:03.300 --> 00:13:05.980
What is your percentile
rank of everyone?

00:13:05.980 --> 00:13:08.620
And they ask basically everyone.

00:13:08.620 --> 00:13:11.030
So you would think
these being engineers,

00:13:11.030 --> 00:13:13.660
engineers being math
guys, that they have

00:13:13.660 --> 00:13:15.140
a good sense of percentiles.

00:13:15.140 --> 00:13:18.160
You'd think that they
might get this right.

00:13:18.160 --> 00:13:19.982
The average engineer
ranks themself right

00:13:19.982 --> 00:13:21.190
about at the 80th percentile.

00:13:23.930 --> 00:13:25.657
Of all the people
they asked-- they

00:13:25.657 --> 00:13:27.990
asked, like, maybe a hundred
of them-- of all the people

00:13:27.990 --> 00:13:30.850
they asked, two ranked
themselves as below average.

00:13:30.850 --> 00:13:33.310
That is my favorite data
point from this study.

00:13:33.310 --> 00:13:36.620
Two ranked themselves
as below average.

00:13:36.620 --> 00:13:37.550
Should be about half.

00:13:40.076 --> 00:13:41.450
And they're not
alone too, right?

00:13:41.450 --> 00:13:44.881
So poker players also famous
for their self-assessment bias.

00:13:44.881 --> 00:13:46.630
And one of the things
that I think sort of

00:13:46.630 --> 00:13:48.080
feeds into this,
at the beginning

00:13:48.080 --> 00:13:49.700
of Rounders-- I haven't seen
this movie for, like, five

00:13:49.700 --> 00:13:49.820
years.

00:13:49.820 --> 00:13:51.910
I don't know if you
guys have seen Rounders.

00:13:51.910 --> 00:13:53.374
But at the beginning
of Rounders,

00:13:53.374 --> 00:13:55.290
Matt Damon's character
quotes this poker book.

00:13:55.290 --> 00:13:56.915
It's one of my all
time favorite quotes

00:13:56.915 --> 00:14:00.230
from a poker movie which,
I guess, isn't saying much.

00:14:00.230 --> 00:14:02.740
But gives this
quote, "Few players

00:14:02.740 --> 00:14:04.590
can remember the big
pots they've won,

00:14:04.590 --> 00:14:07.640
but every player can remember
with remarkable accuracy

00:14:07.640 --> 00:14:10.430
the outstanding tough
beats of their career."

00:14:10.430 --> 00:14:12.310
And I think it's these
sort of memory biases

00:14:12.310 --> 00:14:14.690
that feed into our
self-assessment bias, right?

00:14:14.690 --> 00:14:18.350
So you walk into the Bellagio,
you see a table of all pros.

00:14:18.350 --> 00:14:20.960
There has to be self-assessment
bias here, right?

00:14:20.960 --> 00:14:22.840
There has to be.

00:14:22.840 --> 00:14:26.479
So don't be that person.

00:14:26.479 --> 00:14:27.770
Let me circle back to my point.

00:14:27.770 --> 00:14:29.645
I'm going to tell you
about my favorite poker

00:14:29.645 --> 00:14:30.570
hand of all time.

00:14:30.570 --> 00:14:32.710
Of every hand I've
ever played, this

00:14:32.710 --> 00:14:34.800
is my favorite for
a number of reasons.

00:14:34.800 --> 00:14:36.800
So I'm playing a tournament
in the World Series.

00:14:36.800 --> 00:14:38.020
This is a big tournament.

00:14:38.020 --> 00:14:40.380
And we're on the
exact money bubble.

00:14:40.380 --> 00:14:43.370
And what that means is that
the next person to bust

00:14:43.370 --> 00:14:44.620
out gets nothing.

00:14:44.620 --> 00:14:47.360
Everyone else is guaranteed
something, couple dozen bucks.

00:14:47.360 --> 00:14:48.890
I don't know.

00:14:48.890 --> 00:14:53.780
So I'm sitting there, and
there's this guy at my table

00:14:53.780 --> 00:14:55.640
who, as we approach
the money bubble,

00:14:55.640 --> 00:14:58.120
he's so excited to be there.

00:14:58.120 --> 00:15:00.390
It's his first World
Series tournament,

00:15:00.390 --> 00:15:01.810
he's about to make the money.

00:15:01.810 --> 00:15:05.190
He's calling his girlfriend
and his buddies every hour

00:15:05.190 --> 00:15:07.720
telling them, yeah, I think
we're almost in the money.

00:15:07.720 --> 00:15:08.860
He's so excited.

00:15:08.860 --> 00:15:10.790
And then it gets
to the point where,

00:15:10.790 --> 00:15:13.409
like, 40 minutes before
the bubble, he just leaves.

00:15:13.409 --> 00:15:14.700
He doesn't want to play a hand.

00:15:14.700 --> 00:15:19.660
He just leaves the table so
he doesn't have to bust out.

00:15:19.660 --> 00:15:21.440
So he comes back.

00:15:21.440 --> 00:15:25.890
And it's now the actual bubble.

00:15:25.890 --> 00:15:27.200
He's to my direct left.

00:15:27.200 --> 00:15:29.520
I'm in the small blind,
he's in the big blind,

00:15:29.520 --> 00:15:31.325
and it folds around to me.

00:15:31.325 --> 00:15:33.200
And at this point, I
feel like there are just

00:15:33.200 --> 00:15:34.790
dollar signs in my eyes, right?

00:15:34.790 --> 00:15:35.670
I'm so excited.

00:15:35.670 --> 00:15:39.590
So I look down and I
see four deuce off suit.

00:15:39.590 --> 00:15:42.460
And we both have
medium stacks, but I

00:15:42.460 --> 00:15:45.120
have this hell of a read on
this guy, so I shove all in.

00:15:47.630 --> 00:15:52.650
He says, I knew you
were going to do that.

00:15:52.650 --> 00:15:54.355
I call blind.

00:15:57.100 --> 00:15:59.120
He calls without
looking at his cards

00:15:59.120 --> 00:16:01.137
because he knew I was
going to shove in.

00:16:01.137 --> 00:16:03.220
It's like the strongest
[INAUDIBLE] I've ever had.

00:16:03.220 --> 00:16:04.210
He flips over his hand.

00:16:04.210 --> 00:16:05.830
He has nine deuce off the suit.

00:16:05.830 --> 00:16:09.750
He has me dominated
with a terrible hand.

00:16:09.750 --> 00:16:11.710
Let's think about this.

00:16:11.710 --> 00:16:14.540
How terrible does
my read have to be?

00:16:14.540 --> 00:16:15.832
How far off does it have to be?

00:16:15.832 --> 00:16:17.873
What has to happen here
for him to call with nine

00:16:17.873 --> 00:16:18.600
deuce off suit.

00:16:18.600 --> 00:16:20.730
Not only does he
have to not care

00:16:20.730 --> 00:16:23.720
about bursting out,
busting out on the bubble.

00:16:23.720 --> 00:16:26.790
He has to not care so
much that he won't even

00:16:26.790 --> 00:16:28.770
look at his cards.

00:16:28.770 --> 00:16:32.540
If he looks at his
hand, he has to fold.

00:16:32.540 --> 00:16:36.809
So he proceeds to win the hand.

00:16:36.809 --> 00:16:39.100
And they count us down, and
he has me literally covered

00:16:39.100 --> 00:16:40.840
by one or two chips.

00:16:40.840 --> 00:16:45.075
And I walk away thinking
I have to go home.

00:16:45.075 --> 00:16:46.950
We rent a math house
every year out in Vegas.

00:16:46.950 --> 00:16:49.040
Me and Bill Chen,
Jerrod Ankenman,

00:16:49.040 --> 00:16:50.630
Mike who's sitting
right over there,

00:16:50.630 --> 00:16:51.420
and a bunch of other of us.

00:16:51.420 --> 00:16:53.810
I have to go home and I
have to tell these guys what

00:16:53.810 --> 00:16:55.226
just-- Kenny, who's
right here-- I

00:16:55.226 --> 00:16:57.260
have to tell these guys
what just happened,

00:16:57.260 --> 00:16:59.300
and I still don't
hear the end of it.

00:16:59.300 --> 00:17:01.786
So what does this mean?

00:17:01.786 --> 00:17:03.410
So I realize the
beginning of this talk

00:17:03.410 --> 00:17:05.834
sounds a little bit like
a commercial for like,

00:17:05.834 --> 00:17:07.050
just be humble.

00:17:07.050 --> 00:17:08.859
Don't be an idiot.

00:17:08.859 --> 00:17:11.670
But I think it's a
little more than that.

00:17:11.670 --> 00:17:13.329
Here's the takeaway.

00:17:13.329 --> 00:17:17.800
Any time you try to divine
your opponent's strategy,

00:17:17.800 --> 00:17:21.619
she can do the same
thing back to you.

00:17:21.619 --> 00:17:23.990
And this happens in
really obvious ways,

00:17:23.990 --> 00:17:30.660
like when my super tight guy
all of a sudden doesn't care.

00:17:30.660 --> 00:17:33.190
But it also happens
in subtle ways, right?

00:17:33.190 --> 00:17:35.100
So a lot of times we
think about pot odds.

00:17:35.100 --> 00:17:37.030
So you might be
sitting there thinking,

00:17:37.030 --> 00:17:40.060
OK, I'm getting two to one.

00:17:40.060 --> 00:17:43.800
Do I have the best hand here
at least one in three times?

00:17:43.800 --> 00:17:46.280
Very natural way
to think, right?

00:17:46.280 --> 00:17:48.105
But now you're kind
of trying to do magic.

00:17:48.105 --> 00:17:49.980
That's one of my favorite
quotes of all time,

00:17:49.980 --> 00:17:51.590
by the way, from Harry Potter.

00:17:51.590 --> 00:17:53.795
And the trouble is, the
other side can do magic too.

00:17:53.795 --> 00:17:55.920
I'm trying to figure out
what their distribution is

00:17:55.920 --> 00:17:57.470
and respond to it.

00:17:57.470 --> 00:18:00.280
They're trying to
figure out what

00:18:00.280 --> 00:18:03.220
I think their distribution
is and shape it around that.

00:18:03.220 --> 00:18:05.880
And all of a sudden, we're
playing this leveling game.

00:18:05.880 --> 00:18:10.140
And we're both kind of
trying to do magic, right?

00:18:10.140 --> 00:18:12.512
So in the hand that I showed
you guys in the beginning,

00:18:12.512 --> 00:18:13.970
I could be sitting
there and trying

00:18:13.970 --> 00:18:16.680
to figure out exactly what
my opponent's range is

00:18:16.680 --> 00:18:20.680
when she bets one and
a half times the pot.

00:18:20.680 --> 00:18:24.250
I can try to figure out why
he's not putting me all in.

00:18:24.250 --> 00:18:26.637
I can try to figure out a
million other little things

00:18:26.637 --> 00:18:28.220
that are going to
help me get a little

00:18:28.220 --> 00:18:31.957
more sense of this range,
but I'm trying to do magic.

00:18:31.957 --> 00:18:34.540
And I think that's kind of our
go to move when we don't really

00:18:34.540 --> 00:18:37.830
have a repertoire, right?

00:18:37.830 --> 00:18:40.157
And this is going to happen
more and more-- well,

00:18:40.157 --> 00:18:41.615
it happens with
really bad players,

00:18:41.615 --> 00:18:43.448
bu that's OK-- but it's
going to happen more

00:18:43.448 --> 00:18:44.845
and more with good players.

00:18:44.845 --> 00:18:46.220
They'll take you
to a place where

00:18:46.220 --> 00:18:48.680
they have this paved road.

00:18:48.680 --> 00:18:50.140
They've been there before.

00:18:50.140 --> 00:18:53.110
They take people to
these weird places.

00:18:53.110 --> 00:18:54.570
And you haven't.

00:18:54.570 --> 00:18:57.310
And what do you do?

00:18:57.310 --> 00:19:03.530
So what I would say
is forget their hand.

00:19:03.530 --> 00:19:05.765
Forget their range.

00:19:05.765 --> 00:19:06.640
Don't think about it.

00:19:09.750 --> 00:19:14.084
So one of my favorite people
in the world, Jerrod Ankenman,

00:19:14.084 --> 00:19:15.500
he's the one who
wrote Bill's book

00:19:15.500 --> 00:19:18.060
and let Bill put his
name on it, by the way.

00:19:18.060 --> 00:19:22.430
He has this quote, "If I
truly played optimally,

00:19:22.430 --> 00:19:24.860
I could write down
my entire strategy

00:19:24.860 --> 00:19:26.560
on a piece of
paper, what I would

00:19:26.560 --> 00:19:29.340
do in every single situation,
and I could give it to you

00:19:29.340 --> 00:19:31.920
and you couldn't beat me."

00:19:31.920 --> 00:19:34.080
That's what we're
trying to do here.

00:19:34.080 --> 00:19:36.540
So how do we get there?

00:19:36.540 --> 00:19:39.860
Well, heads-up limit hold'em
turns out to be a pretty big

00:19:39.860 --> 00:19:40.360
game.

00:19:40.360 --> 00:19:42.570
It has about a
quintillion game states,

00:19:42.570 --> 00:19:44.590
and that's the
smallest poker game

00:19:44.590 --> 00:19:46.550
that we really play for money.

00:19:46.550 --> 00:19:49.570
So we can do a
couple things, right?

00:19:49.570 --> 00:19:53.660
We can try to do some, like,
sweet programming like the team

00:19:53.660 --> 00:19:55.422
out in Alberta did.

00:19:55.422 --> 00:19:56.130
And that's great.

00:20:00.000 --> 00:20:03.790
You know, for example,
humans play chess, right?

00:20:03.790 --> 00:20:07.550
And computers now crush them,
but human learning is really

00:20:07.550 --> 00:20:10.340
strongly aided by computers.

00:20:10.340 --> 00:20:12.300
But you can't memorize
every position.

00:20:12.300 --> 00:20:14.550
You can't memorize every line.

00:20:14.550 --> 00:20:17.650
You have to know what's
going on underneath the hood.

00:20:17.650 --> 00:20:20.710
And these algorithms, these
programs like Cepheus, they

00:20:20.710 --> 00:20:22.380
will give you the strategy.

00:20:22.380 --> 00:20:24.440
They will not tell you why.

00:20:24.440 --> 00:20:27.570
And as humans, we need to
start to understand why

00:20:27.570 --> 00:20:30.870
if we have any hope of carrying
these strategies with us

00:20:30.870 --> 00:20:33.990
and actually playing
them in real life.

00:20:33.990 --> 00:20:35.587
So they're a black box, right?

00:20:35.587 --> 00:20:36.670
So we can solve the games.

00:20:36.670 --> 00:20:40.720
But solving the games does
not get us out of the woods.

00:20:40.720 --> 00:20:44.460
So if we want to wrap our
puny human minds around it,

00:20:44.460 --> 00:20:46.310
we have to be a
little more clever.

00:20:46.310 --> 00:20:49.340
So we're going to look at
a couple of toy games here,

00:20:49.340 --> 00:20:50.880
really simple ones
that can start

00:20:50.880 --> 00:20:53.620
to help us wrap our
minds around how

00:20:53.620 --> 00:20:55.780
to behave in these situations.

00:20:55.780 --> 00:20:58.105
So the first one is
a clairvoyance game.

00:20:58.105 --> 00:21:00.970
And a clairvoyance
game is basically

00:21:00.970 --> 00:21:04.920
when either one or both players
have complete game state

00:21:04.920 --> 00:21:06.130
information.

00:21:06.130 --> 00:21:08.370
So what would you do
if you lived in a world

00:21:08.370 --> 00:21:10.960
where you always knew
your opponent's hand

00:21:10.960 --> 00:21:12.565
and he knew that you knew?

00:21:12.565 --> 00:21:14.190
That's the idea of
a clairvoyance game.

00:21:14.190 --> 00:21:17.344
So we're going to look at a game
called Coin Flip Clairvoyance.

00:21:17.344 --> 00:21:18.510
And the game goes like this.

00:21:18.510 --> 00:21:21.225
You can and should play
it for money by the way.

00:21:21.225 --> 00:21:22.954
So each player antes a dollar.

00:21:22.954 --> 00:21:23.870
There are two players.

00:21:23.870 --> 00:21:24.911
And then you flip a coin.

00:21:24.911 --> 00:21:26.820
If it's heads, you win.

00:21:26.820 --> 00:21:28.970
If it's tails,
your opponent wins.

00:21:28.970 --> 00:21:31.950
However, there's a
round of betting.

00:21:31.950 --> 00:21:34.350
Only you see the
coin after the flip.

00:21:34.350 --> 00:21:37.370
Then you can bet or check.

00:21:37.370 --> 00:21:39.520
So you choose to bet
a dollar or check,

00:21:39.520 --> 00:21:41.730
and your opponent
can only check.

00:21:41.730 --> 00:21:44.340
If you check, they have
to flip over hand or call.

00:21:44.340 --> 00:21:49.040
If you bet, they either have
to call a dollar or fold, OK?

00:21:49.040 --> 00:21:51.382
So you know if you win or not.

00:21:51.382 --> 00:21:53.590
Your opponent doesn't know
if they'll win a showdown.

00:21:53.590 --> 00:21:54.798
That's the idea of this game.

00:21:54.798 --> 00:21:56.650
Make sense?

00:21:56.650 --> 00:21:58.780
Yeah OK, easy game.

00:21:58.780 --> 00:22:00.250
So how do we play?

00:22:00.250 --> 00:22:02.110
So scenario one.

00:22:02.110 --> 00:22:02.860
Flip the coin.

00:22:02.860 --> 00:22:03.840
It's heads.

00:22:03.840 --> 00:22:05.773
What do we do?

00:22:05.773 --> 00:22:06.715
AUDIENCE: Bet.

00:22:06.715 --> 00:22:07.657
MATT HAWRILENKO: Bet.

00:22:07.657 --> 00:22:08.600
OK.

00:22:08.600 --> 00:22:09.700
Scenario two.

00:22:09.700 --> 00:22:10.950
It's tails.

00:22:10.950 --> 00:22:12.180
What do we do?

00:22:12.180 --> 00:22:14.100
So there are two main questions.

00:22:14.100 --> 00:22:16.720
How often should
your opponent call,

00:22:16.720 --> 00:22:19.130
and how often should you bluff?

00:22:19.130 --> 00:22:20.710
So how do we solve this?

00:22:20.710 --> 00:22:23.700
What we want to do
is we want to call

00:22:23.700 --> 00:22:26.870
enough to make your opponent
indifferent to bluffing

00:22:26.870 --> 00:22:27.480
or giving up.

00:22:27.480 --> 00:22:29.230
That's what your
opponent should be doing.

00:22:29.230 --> 00:22:30.930
So to do that,
we're going to set

00:22:30.930 --> 00:22:34.680
the expectation of bluffing
equal to the expectation

00:22:34.680 --> 00:22:35.390
of giving up.

00:22:35.390 --> 00:22:38.510
So the expectation of
bluffing is just this, right?

00:22:38.510 --> 00:22:41.084
It's the pot times
the amount they fold,

00:22:41.084 --> 00:22:43.000
which is one minus the
proportion of the times

00:22:43.000 --> 00:22:44.040
they called, right?

00:22:44.040 --> 00:22:46.290
That's how much you
win when you bluff.

00:22:46.290 --> 00:22:48.400
So now how much do
you lose when you

00:22:48.400 --> 00:22:50.250
bluff and get caught, right?

00:22:50.250 --> 00:22:52.170
It's the amount
you bluff, which is

00:22:52.170 --> 00:22:56.420
going to be one unit, times
the amount you get called.

00:22:56.420 --> 00:23:00.880
So we can just sort of do
a little simple algebra,

00:23:00.880 --> 00:23:03.830
and it'll reduce to-- you
should be calling p over p

00:23:03.830 --> 00:23:07.030
plus 1 of the time, where
p is the pot, right?

00:23:07.030 --> 00:23:11.550
So if the pot is two units
and you're bluffing one unit,

00:23:11.550 --> 00:23:15.570
we should be calling with a
proportion of two over three.

00:23:15.570 --> 00:23:17.260
2/3 of the time with our kings.

00:23:17.260 --> 00:23:20.180
That is how we make the
opponent indifferent to bluffing

00:23:20.180 --> 00:23:21.428
or giving up.

00:23:21.428 --> 00:23:22.640
OK?

00:23:22.640 --> 00:23:26.250
So a couple of things to note
about this before we push on.

00:23:26.250 --> 00:23:28.930
So what happens here
as the pot gets bigger?

00:23:28.930 --> 00:23:31.294
Do you call more or less?

00:23:31.294 --> 00:23:32.074
AUDIENCE: More

00:23:32.074 --> 00:23:32.990
MATT HAWRILENKO: More.

00:23:32.990 --> 00:23:36.469
Yeah/ So as the pot gets bigger,
this asymptotes to 1 which

00:23:36.469 --> 00:23:37.260
makes sense, right?

00:23:37.260 --> 00:23:38.884
Most of the value's
already in the pot.

00:23:38.884 --> 00:23:40.780
So there's more money in there.

00:23:40.780 --> 00:23:44.290
You have to protect against
being bluffed at more.

00:23:44.290 --> 00:23:47.290
Totally intuitive
with poker, right?

00:23:47.290 --> 00:23:48.634
So how often do you bluff?

00:23:48.634 --> 00:23:50.300
Well, you want to set
the expected value

00:23:50.300 --> 00:23:53.416
of calling equal to the
expected value of folding.

00:23:53.416 --> 00:23:55.257
So the expected
value of calling here

00:23:55.257 --> 00:23:57.840
is just-- we're going to use the
ratio of bluffs to value bets

00:23:57.840 --> 00:23:58.881
rather than a percentage.

00:23:58.881 --> 00:24:00.610
It just works out nicer.

00:24:00.610 --> 00:24:04.520
So the ratio of bluffs to
value bets, so how frequently

00:24:04.520 --> 00:24:08.250
are you bluffing,
times the pot plus one

00:24:08.250 --> 00:24:11.946
because that is what
they win when they call.

00:24:11.946 --> 00:24:13.945
So bluffs value that times
the pot plus the unit

00:24:13.945 --> 00:24:16.020
that you bluff.

00:24:16.020 --> 00:24:19.285
And then what do they lose
when they call in their wrong?

00:24:19.285 --> 00:24:20.910
They lose the amount
you bet, so that's

00:24:20.910 --> 00:24:23.450
going to be one unit,
the value you bet.

00:24:23.450 --> 00:24:26.400
So you're going
to bluff p plus 1.

00:24:26.400 --> 00:24:29.480
So you're going to bluff 1
over p plus 1 of the time.

00:24:29.480 --> 00:24:31.800
So now what happens here?

00:24:31.800 --> 00:24:33.850
So as the pot gets bigger,
what are you doing?

00:24:33.850 --> 00:24:35.880
Bluffing more or less?

00:24:35.880 --> 00:24:37.004
AUDIENCE: Less.

00:24:37.004 --> 00:24:37.920
MATT HAWRILENKO: Less.

00:24:37.920 --> 00:24:38.590
Yeah.

00:24:38.590 --> 00:24:41.370
Is that counterintuitive?

00:24:41.370 --> 00:24:42.280
AUDIENCE: No.

00:24:42.280 --> 00:24:42.640
MATT HAWRILENKO: Is it?

00:24:42.640 --> 00:24:43.182
I don't know.

00:24:43.182 --> 00:24:45.640
It was counterintuitive to me
because I'm like, oh, there's

00:24:45.640 --> 00:24:46.560
more money in the pot.

00:24:46.560 --> 00:24:50.150
But what it means is there's
more money in the pot,

00:24:50.150 --> 00:24:53.270
so I don't really need to bluff
very frequently to make sure

00:24:53.270 --> 00:24:55.640
I get value, because
the values in there,

00:24:55.640 --> 00:25:00.180
and my opponent is
calling more of the time.

00:25:00.180 --> 00:25:02.875
So the bigger the pot is, the
more my opponent calls, right?

00:25:02.875 --> 00:25:04.250
And what I'm doing
essentially is

00:25:04.250 --> 00:25:07.140
I'm bluffing so I can get value
from the time that I'm winning.

00:25:07.140 --> 00:25:07.640
Yeah?

00:25:11.262 --> 00:25:12.110
OK.

00:25:12.110 --> 00:25:14.026
So we can actually
generalize this too, right?

00:25:14.026 --> 00:25:16.907
So we can generalize it to
no limit games pretty simply.

00:25:16.907 --> 00:25:18.740
So we've sort of flipped
things around here.

00:25:18.740 --> 00:25:22.035
So here the pot is 1 and s
is the proportion of the pot

00:25:22.035 --> 00:25:25.000
that you bet, so you're
going to be calling 1 over 1

00:25:25.000 --> 00:25:26.050
plus s, right?

00:25:26.050 --> 00:25:31.030
So if the pot is two
units, s is one unit,

00:25:31.030 --> 00:25:32.410
so s would be 0.5, right?

00:25:32.410 --> 00:25:36.570
1 plus 0.5 is one half.

00:25:36.570 --> 00:25:39.140
And you'd be bluffing
s over 1 plus s, right?

00:25:39.140 --> 00:25:42.230
So you're going to be calling
1 minus the bluff ratio.

00:25:42.230 --> 00:25:44.105
Sometimes we call the
bluff ratio alpha.

00:25:44.105 --> 00:25:47.270
1 minus alpha.

00:25:47.270 --> 00:25:49.400
But we can actually calculate
a value for this game

00:25:49.400 --> 00:25:50.830
and for all toy games.

00:25:50.830 --> 00:25:53.110
So Bowling and their team
with Cepheus they calculated

00:25:53.110 --> 00:25:55.550
the value of having the
button in limit hold'em.

00:25:55.550 --> 00:25:57.070
We always kind of
knew it was good,

00:25:57.070 --> 00:25:59.195
but they calculated precisely
just how good it was.

00:25:59.195 --> 00:26:01.569
And if you're looking for a
job at a place like [? Sig ?]

00:26:01.569 --> 00:26:03.360
or somewhere else in
finance, you actually

00:26:03.360 --> 00:26:05.480
probably should just
calculate the value of this,

00:26:05.480 --> 00:26:06.896
because you're
going to be getting

00:26:06.896 --> 00:26:10.100
interview questions like this.

00:26:10.100 --> 00:26:14.600
So again, what we're taking
here is the larger amount

00:26:14.600 --> 00:26:18.720
your opponent gets, the less
frequently you have to call.

00:26:18.720 --> 00:26:19.550
Right?

00:26:19.550 --> 00:26:22.040
The more frequently
your opponent bluffs,

00:26:22.040 --> 00:26:25.450
the more frequently she
has to value that, right?

00:26:25.450 --> 00:26:29.130
So as a bluffing
region gets larger,

00:26:29.130 --> 00:26:31.707
the value betting region
has to get larger with it.

00:26:31.707 --> 00:26:33.540
So in a coin flip game
it doesn't make sense

00:26:33.540 --> 00:26:37.655
to think of regions, but
in poker it's going to.

00:26:37.655 --> 00:26:39.280
So a question I get
here a lot is, what

00:26:39.280 --> 00:26:40.700
if it's not a repeated game?

00:26:40.700 --> 00:26:42.810
What if you're playing
it just this one time?

00:26:42.810 --> 00:26:44.930
Or what if you're at
a table with a player

00:26:44.930 --> 00:26:47.080
that you'll never
play with again?

00:26:47.080 --> 00:26:49.640
How do you play that?

00:26:49.640 --> 00:26:52.570
And the answer is that
it's a repeated game.

00:26:52.570 --> 00:26:57.480
So I have a feeling this will
be intuitive for you guys,

00:26:57.480 --> 00:26:59.160
so I'll do it quickly.

00:26:59.160 --> 00:27:01.040
But I want you to
imagine a scenario

00:27:01.040 --> 00:27:03.248
where you're going to play
the Coin Flip Clairvoyance

00:27:03.248 --> 00:27:06.260
game a thousand times
against different players.

00:27:06.260 --> 00:27:09.340
So against a thousand
different players, right?

00:27:09.340 --> 00:27:11.550
And suppose you
take the position,

00:27:11.550 --> 00:27:13.910
since my opponents can't
learn from my past,

00:27:13.910 --> 00:27:16.710
I'm going to bluff
100% of the time.

00:27:16.710 --> 00:27:19.180
So what's actually
happening here?

00:27:19.180 --> 00:27:23.170
You can sort of think of each
opponent as a random sampling

00:27:23.170 --> 00:27:26.089
from the distribution
of possible strategies

00:27:26.089 --> 00:27:28.380
that are out there for the
Coin Flip Clairvoyance game.

00:27:28.380 --> 00:27:32.070
So some of them will fold too
much, and you will own them.

00:27:32.070 --> 00:27:34.700
And then some of them
will call too much,

00:27:34.700 --> 00:27:37.900
and they will own you if you're
bluffing all of the time,

00:27:37.900 --> 00:27:38.460
right?

00:27:38.460 --> 00:27:40.920
So even if you haven't
seen this person before,

00:27:40.920 --> 00:27:43.470
even if it's your first
hand of poker against them,

00:27:43.470 --> 00:27:46.130
it is a repeated game, OK?

00:27:46.130 --> 00:27:49.200
So get that, it's not a repeated
game notion out of your heads.

00:27:49.200 --> 00:27:52.320
It's bad for business.

00:27:52.320 --> 00:27:54.165
So taking like a
half a step back,

00:27:54.165 --> 00:27:55.790
what have we learned
about this so far?

00:27:55.790 --> 00:27:56.420
OK.

00:27:56.420 --> 00:28:00.020
So the Coin Flip Clairvoyance
game, it's not about just value

00:28:00.020 --> 00:28:01.840
betting or just bluffing.

00:28:01.840 --> 00:28:04.390
It's about the
combination of the two.

00:28:04.390 --> 00:28:07.110
We're also trying to maximize
the value of our entire set

00:28:07.110 --> 00:28:08.400
of hands, right?

00:28:08.400 --> 00:28:09.620
Because what happens?

00:28:09.620 --> 00:28:12.190
So suppose our
strategy is, we're

00:28:12.190 --> 00:28:13.970
going to bet every
time we have heads,

00:28:13.970 --> 00:28:16.020
we're never going to
bet when we have tails?

00:28:16.020 --> 00:28:18.984
What's our opponent going to do?

00:28:18.984 --> 00:28:20.150
AUDIENCE: Fold to every bet.

00:28:20.150 --> 00:28:22.316
MATT HAWRILENKO: They're
going to fold to every bet.

00:28:22.316 --> 00:28:24.240
Yeah, exactly.

00:28:24.240 --> 00:28:27.230
So we sort of can
calculate this ratio

00:28:27.230 --> 00:28:31.985
where now they don't do so
well if they fold to every bet.

00:28:31.985 --> 00:28:34.360
And so what's cool about this
is the math in this game is

00:28:34.360 --> 00:28:35.151
very simple, right?

00:28:35.151 --> 00:28:35.870
It's not hard.

00:28:35.870 --> 00:28:37.750
But it buys you a lot.

00:28:37.750 --> 00:28:41.690
It buys you a lot of
intuition about poker.

00:28:41.690 --> 00:28:44.660
Some really useful concepts.

00:28:44.660 --> 00:28:46.890
So I want to move on.

00:28:46.890 --> 00:28:49.310
This is probably my favorite
of all the toy games,

00:28:49.310 --> 00:28:51.310
and there are actually a
million versions of it.

00:28:51.310 --> 00:28:52.726
We're going to do
the simplest one

00:28:52.726 --> 00:28:57.460
because I think it sort of
gets everything that you

00:28:57.460 --> 00:28:58.914
need to know, more or less.

00:28:58.914 --> 00:29:00.580
So this is an ace,
king, and queen game.

00:29:00.580 --> 00:29:02.870
It's an incomplete
information game.

00:29:02.870 --> 00:29:06.290
So each player antes $1
and is dealt one card.

00:29:06.290 --> 00:29:07.820
So if I get delta
ace, my opponent

00:29:07.820 --> 00:29:09.700
only has the king or the queen.

00:29:09.700 --> 00:29:12.070
They can't also have an ace.

00:29:12.070 --> 00:29:14.340
So then you can check of
bet and your opponent,

00:29:14.340 --> 00:29:16.090
just like the Coin
Flip Clairvoyance game,

00:29:16.090 --> 00:29:20.060
can only check or call or fold.

00:29:20.060 --> 00:29:22.760
They can't bet if you check.

00:29:22.760 --> 00:29:26.480
So this is going to be our
first mapping, our only mapping,

00:29:26.480 --> 00:29:30.880
of a toy game that
resembles real poker, right?

00:29:30.880 --> 00:29:33.960
Now we have a real range.

00:29:33.960 --> 00:29:36.620
Again, I think that you
probably can and should

00:29:36.620 --> 00:29:37.662
play this game for money.

00:29:37.662 --> 00:29:39.828
I think there's a real
difference between what we're

00:29:39.828 --> 00:29:41.960
about to do, which is
solve it and sort of get it

00:29:41.960 --> 00:29:44.570
intuitively, and actually
get it experientially.

00:29:44.570 --> 00:29:46.930
So go forth and gamble.

00:29:46.930 --> 00:29:49.280
But what do we do?

00:29:49.280 --> 00:29:50.250
So case one.

00:29:50.250 --> 00:29:51.430
So you get the ace.

00:29:51.430 --> 00:29:53.443
Are you going to check or bet?

00:29:53.443 --> 00:29:54.026
AUDIENCE: Bet.

00:29:54.026 --> 00:29:55.567
MATT HAWRILENKO:
You're going to bet.

00:29:55.567 --> 00:29:56.970
Yeah.

00:29:56.970 --> 00:30:00.689
Now your opponent bets
and you have an ace.

00:30:00.689 --> 00:30:01.730
What are you going to do?

00:30:01.730 --> 00:30:02.510
AUDIENCE: Call.

00:30:02.510 --> 00:30:04.093
MATT HAWRILENKO:
You're going to call.

00:30:04.093 --> 00:30:05.050
All right, good.

00:30:05.050 --> 00:30:06.790
And now your opponent
bets, and you have a queen.

00:30:06.790 --> 00:30:07.745
What are you going to do?

00:30:07.745 --> 00:30:08.220
AUDIENCE: Fold.

00:30:08.220 --> 00:30:09.136
MATT HAWRILENKO: Yeah.

00:30:09.136 --> 00:30:09.700
Hey, yeah.

00:30:09.700 --> 00:30:10.200
OK.

00:30:10.200 --> 00:30:11.690
So it seems trivial, right?

00:30:11.690 --> 00:30:14.365
These first three cases
seem really trivial.

00:30:14.365 --> 00:30:15.740
But an important
thing to note is

00:30:15.740 --> 00:30:19.430
that they are
dominant strategies

00:30:19.430 --> 00:30:20.880
or dominated strategies.

00:30:20.880 --> 00:30:23.570
So a dominated strategy in
game theory, for example,

00:30:23.570 --> 00:30:26.325
calling with a queen
here would be dominated.

00:30:26.325 --> 00:30:30.390
A dominated strategy is
one where the decision has

00:30:30.390 --> 00:30:33.240
equal or lesser-- strictly
equal or lesser value

00:30:33.240 --> 00:30:34.580
to another decision.

00:30:34.580 --> 00:30:37.370
If I call with a
queen, I cannot win.

00:30:37.370 --> 00:30:38.700
I just lose money.

00:30:38.700 --> 00:30:42.450
That has strictly lesser
value than folding, OK?

00:30:42.450 --> 00:30:45.360
So dominated strategies,
important concept.

00:30:45.360 --> 00:30:47.540
So how about this one.

00:30:47.540 --> 00:30:48.630
You have a king.

00:30:48.630 --> 00:30:50.148
Check or bet?

00:30:50.148 --> 00:30:50.945
AUDIENCE: Check.

00:30:50.945 --> 00:30:51.570
AUDIENCE: Check

00:30:51.570 --> 00:30:52.000
MATT HAWRILENKO: Check.

00:30:52.000 --> 00:30:52.666
AUDIENCE: Split.

00:30:52.666 --> 00:30:53.710
MATT HAWRILENKO: Split?

00:30:53.710 --> 00:30:54.210
OK.

00:30:54.210 --> 00:30:57.599
All right, well, I actually
want to see where everyone's in.

00:30:57.599 --> 00:30:59.140
So we're going to
have three options.

00:30:59.140 --> 00:31:01.126
Who wants to check?

00:31:01.126 --> 00:31:05.540
All right, who wants
to bet all the time?

00:31:05.540 --> 00:31:08.090
Who wants to bet sometimes?

00:31:08.090 --> 00:31:08.965
OK.

00:31:08.965 --> 00:31:10.630
See, I tricked you.

00:31:10.630 --> 00:31:14.420
This is also a
dominated strategy.

00:31:14.420 --> 00:31:16.880
So what happens if you
have a king and you bet?

00:31:16.880 --> 00:31:18.910
What is your opponent
going to do with an ace?

00:31:18.910 --> 00:31:19.980
AUDIENCE: He's either
going to call with an ace

00:31:19.980 --> 00:31:21.050
or fold with a queen.

00:31:21.050 --> 00:31:22.091
MATT HAWRILENKO: Exactly.

00:31:22.091 --> 00:31:24.550
Your opponent-- because
the ace and the queen,

00:31:24.550 --> 00:31:26.260
dominant strategies, right?

00:31:26.260 --> 00:31:28.060
Your opponents always
calling with an ace,

00:31:28.060 --> 00:31:29.620
always folding a queen.

00:31:29.620 --> 00:31:34.700
So betting with a king here
would be a dominated strategy.

00:31:34.700 --> 00:31:37.580
Strictly dominated by checking.

00:31:37.580 --> 00:31:38.730
How about here?

00:31:38.730 --> 00:31:40.814
Now we have a queen.

00:31:40.814 --> 00:31:41.730
What do we want to do?

00:31:41.730 --> 00:31:42.623
Check or bet?

00:31:42.623 --> 00:31:43.449
AUDIENCE: Bet.

00:31:43.449 --> 00:31:44.990
MATT HAWRILENKO:
Check, bet, exactly.

00:31:44.990 --> 00:31:48.121
We want to bet some of the time.

00:31:48.121 --> 00:31:48.620
Right?

00:31:51.135 --> 00:31:52.510
And we'll go
through [INAUDIBLE].

00:31:52.510 --> 00:31:54.540
And now we have a king
and our opponent bets.

00:31:54.540 --> 00:31:56.584
What do we want to do?

00:31:56.584 --> 00:31:57.560
AUDIENCE: Mix.

00:31:57.560 --> 00:31:58.460
MATT HAWRILENKO: Mix.

00:31:58.460 --> 00:31:58.959
Good.

00:31:58.959 --> 00:31:59.880
You got the idea.

00:31:59.880 --> 00:32:04.647
Does anyone have any guess as
to how the mix might break down?

00:32:04.647 --> 00:32:06.105
AUDIENCE: Depends
on his proportion

00:32:06.105 --> 00:32:09.017
of bluffing with the queen.

00:32:09.017 --> 00:32:09.850
MATT HAWRILENKO: OK.

00:32:09.850 --> 00:32:12.046
But suppose we're
trying to solve it.

00:32:12.046 --> 00:32:14.420
So it depends on his proportions
bluffing with the queen.

00:32:14.420 --> 00:32:15.400
Exactly.

00:32:15.400 --> 00:32:16.940
And let's get tighter.

00:32:16.940 --> 00:32:18.464
How do we solve it, right?

00:32:18.464 --> 00:32:19.880
So if we have a
king, our opponent

00:32:19.880 --> 00:32:21.550
has an ace half the time and
has a queen half the time.

00:32:21.550 --> 00:32:23.510
They're going to bet all
the time with the ace

00:32:23.510 --> 00:32:25.516
and sometimes with the queen.

00:32:25.516 --> 00:32:29.000
Turns out in this
game that it's going

00:32:29.000 --> 00:32:32.920
to be about the same
or exactly the same.

00:32:32.920 --> 00:32:37.300
She should be calling
1 over 1 plus s, s

00:32:37.300 --> 00:32:38.900
being the fraction of the pot.

00:32:38.900 --> 00:32:43.670
So you should be calling with
2/3 of hands that beat a bluff.

00:32:43.670 --> 00:32:47.350
So the hands that beat a bluff
are aces and kings, not queens,

00:32:47.350 --> 00:32:47.850
right?

00:32:52.150 --> 00:32:55.000
So if we're thinking about
it this way, before-- eh, no.

00:32:55.000 --> 00:32:56.490
I'll show this first.

00:32:56.490 --> 00:32:59.197
So aces are going to represent
50% of the hands that

00:32:59.197 --> 00:33:00.030
beat a bluff, right?

00:33:00.030 --> 00:33:01.405
Because you're
going to have aces

00:33:01.405 --> 00:33:03.950
50% of the time and
kings 50% of the time.

00:33:03.950 --> 00:33:07.140
So calling with aces seems
better than calling with kings.

00:33:07.140 --> 00:33:09.260
So we're going to call
with all of our aces.

00:33:09.260 --> 00:33:12.180
So now we're up to half, but
we need to get to 2/3, right?

00:33:12.180 --> 00:33:16.100
We want to be calling 2/3 of the
time, kind of per our formula.

00:33:16.100 --> 00:33:18.030
So we're calling
with all of our aces,

00:33:18.030 --> 00:33:20.030
and then a third of
our kings times having

00:33:20.030 --> 00:33:24.440
a king half the time,
that's another sixth, right?

00:33:24.440 --> 00:33:28.320
So all of our aces and
a third of our kings.

00:33:28.320 --> 00:33:32.800
So how is thinking
about it this way

00:33:32.800 --> 00:33:37.540
different from thinking
about it using pot odds?

00:33:37.540 --> 00:33:41.120
So for pot odds we're
trying to figure out,

00:33:41.120 --> 00:33:45.080
what does this person
have in this situation?

00:33:45.080 --> 00:33:47.310
So I'm sitting here with
a king with pot odds,

00:33:47.310 --> 00:33:53.240
and I'm thinking, am I ahead at
least a third of the time here?

00:33:53.240 --> 00:33:54.395
I don't know exactly.

00:33:54.395 --> 00:33:55.770
But I know that
I can try to make

00:33:55.770 --> 00:33:59.110
my opponent indifferent
to bluffing or calling.

00:33:59.110 --> 00:34:00.920
So I'm thinking
about what I'm doing

00:34:00.920 --> 00:34:02.635
with my whole range of hands.

00:34:06.540 --> 00:34:09.655
So yeah, OK.

00:34:09.655 --> 00:34:11.030
So these are the
two observations

00:34:11.030 --> 00:34:12.863
we had from the Coin
Flip Clairvoyance game.

00:34:12.863 --> 00:34:14.326
So adding one.

00:34:14.326 --> 00:34:16.909
One thing that we're noting here
with the ace, king, and queen

00:34:16.909 --> 00:34:17.908
game, what are we doing?

00:34:17.908 --> 00:34:21.480
We're sort of implicitly mapping
three different types of hands.

00:34:21.480 --> 00:34:24.159
Value hands, bluff
catchers, and bluffs.

00:34:27.449 --> 00:34:29.710
And the big thing
here is your strategy

00:34:29.710 --> 00:34:32.870
for what you do with one
hand determines your strategy

00:34:32.870 --> 00:34:34.440
for other hands, all right?

00:34:34.440 --> 00:34:37.010
I'm definitely
calling with the aces,

00:34:37.010 --> 00:34:39.090
so I need to call with
some kings, right?

00:34:39.090 --> 00:34:41.800
I'm definitely
betting all my aces,

00:34:41.800 --> 00:34:45.309
so I need to bluff with the
lowest-- like, the worst

00:34:45.309 --> 00:34:46.600
part of my distribution, right?

00:34:46.600 --> 00:34:50.400
That's the part that's
going to gain the most.

00:34:50.400 --> 00:34:54.679
So a more subtle thing that
I think is super important

00:34:54.679 --> 00:34:57.450
and is going to play into sort
of the last half of this talk

00:34:57.450 --> 00:35:00.830
is if I am playing a hand
differently from you,

00:35:00.830 --> 00:35:04.220
I should do different
things with other hands

00:35:04.220 --> 00:35:05.450
than you should.

00:35:05.450 --> 00:35:10.450
Say for whatever reason I'm
only betting half of my aces

00:35:10.450 --> 00:35:13.440
but I'm still betting
a third of my kings.

00:35:13.440 --> 00:35:14.480
Whoops.

00:35:14.480 --> 00:35:16.370
Now I'm out of whack, right?

00:35:16.370 --> 00:35:19.701
Now I'm out of balance, and I'm
going to lose more in this game

00:35:19.701 --> 00:35:20.700
by being out of balance.

00:35:23.810 --> 00:35:27.620
So your strategy for one
hand determines your strategy

00:35:27.620 --> 00:35:29.480
for other hands.

00:35:29.480 --> 00:35:32.450
That's the whole key here, OK?

00:35:32.450 --> 00:35:35.390
So to sort of summarize
what we've done so far

00:35:35.390 --> 00:35:36.390
is the Temple of Apollo.

00:35:36.390 --> 00:35:38.750
This is where I'd like you
to go to see the oracle.

00:35:38.750 --> 00:35:40.500
Like, you want to go
and get a prediction,

00:35:40.500 --> 00:35:41.830
this is where you would go.

00:35:41.830 --> 00:35:44.080
And walking into
the Temple of Apollo

00:35:44.080 --> 00:35:45.875
back in the day
in ancient Greece,

00:35:45.875 --> 00:35:48.380
I'm wondering if they knew a
thing or two about game theory.

00:35:48.380 --> 00:35:50.421
So there are three
inscriptions above the temple.

00:35:50.421 --> 00:35:53.210
The first one is
know thyself, right?

00:35:53.210 --> 00:35:54.660
Know your own hand.

00:35:54.660 --> 00:35:58.090
Know your own distribution.

00:35:58.090 --> 00:36:00.810
The second is nothing in excess.

00:36:00.810 --> 00:36:03.030
Play with balance.

00:36:03.030 --> 00:36:06.620
So know thyself,
nothing in excess.

00:36:06.620 --> 00:36:10.390
And the last one is make a
pledge and mischief is nigh.

00:36:10.390 --> 00:36:12.732
Yeah, it's really a real
stretch to make that one work,

00:36:12.732 --> 00:36:14.190
so we'll just leave
it at mischief.

00:36:14.190 --> 00:36:15.970
Mischief can sometimes be good.

00:36:15.970 --> 00:36:19.880
So let me be very clear
on how important I

00:36:19.880 --> 00:36:23.310
think this concept of
knowing your own hand,

00:36:23.310 --> 00:36:26.190
knowing where you are in
your own distribution is.

00:36:26.190 --> 00:36:30.190
I think you should not think
about anything else in poker

00:36:30.190 --> 00:36:34.380
until you have bought and paid
for a house by knowing where

00:36:34.380 --> 00:36:37.170
you are in your
own distribution,

00:36:37.170 --> 00:36:38.887
in shaping it to be balanced.

00:36:38.887 --> 00:36:40.220
Don't think about anything else.

00:36:40.220 --> 00:36:43.260
Everything else is
just window dressing

00:36:43.260 --> 00:36:46.570
compared to this concept.

00:36:46.570 --> 00:36:48.260
So here we are again.

00:36:48.260 --> 00:36:50.430
So we're going to go
through this hand, the one

00:36:50.430 --> 00:36:52.555
that I told you about at
the beginning of the talk,

00:36:52.555 --> 00:36:54.429
and we're going to try
to read our own hand.

00:36:54.429 --> 00:36:56.470
So again, we're playing
against this player who's

00:36:56.470 --> 00:36:58.800
better than us,
some stuff happens,

00:36:58.800 --> 00:37:00.360
what do we do on the river?

00:37:00.360 --> 00:37:04.599
How do we think about that from
a game theoretic perspective?

00:37:04.599 --> 00:37:06.140
So there are kind
of three ways again

00:37:06.140 --> 00:37:07.550
that I alluded to
in the beginning

00:37:07.550 --> 00:37:08.900
that people might
think about it, right?

00:37:08.900 --> 00:37:11.240
So the first one is my
hand versus your hand, OK?

00:37:11.240 --> 00:37:12.330
Well, I have aces.

00:37:12.330 --> 00:37:13.510
What do I think he has?

00:37:13.510 --> 00:37:15.280
King queen?

00:37:15.280 --> 00:37:15.780
King 10?

00:37:15.780 --> 00:37:17.765
Maybe he has queen 10?

00:37:17.765 --> 00:37:19.210
Maybe he's bluffing?

00:37:19.210 --> 00:37:20.930
Maybe he has queen nine?

00:37:20.930 --> 00:37:22.784
What is his most likely hand?

00:37:22.784 --> 00:37:24.450
And probably as most
of you have already

00:37:24.450 --> 00:37:26.540
realized-- so Kevin told me
that you'd all played at least

00:37:26.540 --> 00:37:28.040
100 tournaments so
far, I'm guessing

00:37:28.040 --> 00:37:33.140
some a lot more-- it's
really hard to put someone

00:37:33.140 --> 00:37:37.040
on a particular hand,
not particularly useful.

00:37:37.040 --> 00:37:39.620
So the next thing
you might try to do

00:37:39.620 --> 00:37:42.070
is my hand versus
your distribution.

00:37:42.070 --> 00:37:46.790
How are my aces doing
against all the hands

00:37:46.790 --> 00:37:50.060
that you might have given
the actions you've taken?

00:37:50.060 --> 00:37:51.850
I can't really put
you on one hand,

00:37:51.850 --> 00:37:55.990
but I can look at
your actions and see

00:37:55.990 --> 00:38:00.814
what sort of distribution
they might suggest.

00:38:00.814 --> 00:38:02.480
And then the last one
is my distribution

00:38:02.480 --> 00:38:04.070
versus your distribution.

00:38:04.070 --> 00:38:06.260
And this is I can
look at your actions

00:38:06.260 --> 00:38:08.080
and I can look at
my actions, and I

00:38:08.080 --> 00:38:11.470
can try to shape my actions
such that they maximally

00:38:11.470 --> 00:38:13.960
exploit your actions.

00:38:13.960 --> 00:38:16.390
So it's not so much about
what I'm doing with my aces.

00:38:16.390 --> 00:38:20.590
It's about what I'm doing with
all the hands that I'd have,

00:38:20.590 --> 00:38:24.309
where I just happen
to have aces here.

00:38:24.309 --> 00:38:26.600
And this is the style that's
most complimentary to game

00:38:26.600 --> 00:38:27.900
theory.

00:38:27.900 --> 00:38:32.250
So again, we're on this river
and with pot odds with my aces

00:38:32.250 --> 00:38:35.410
I might be thinking, am I good
at least a third of the time

00:38:35.410 --> 00:38:37.215
here?

00:38:37.215 --> 00:38:39.090
But if I'm doing that
I'm trying to do magic,

00:38:39.090 --> 00:38:42.180
because I'm trying to figure
out exactly what he has.

00:38:42.180 --> 00:38:44.660
And I don't think we
need to do that here.

00:38:44.660 --> 00:38:45.540
Right?

00:38:45.540 --> 00:38:48.460
What I want to be thinking
is, well, how much of the time

00:38:48.460 --> 00:38:51.260
do I need to call
to make my opponent

00:38:51.260 --> 00:38:54.910
indifferent to bluffing?

00:38:54.910 --> 00:38:57.466
Any guesses?

00:38:57.466 --> 00:38:57.965
Yeah?

00:38:57.965 --> 00:39:00.290
AUDIENCE: Like 40% of the time?

00:39:00.290 --> 00:39:01.230
MATT HAWRILENKO: Yeah.

00:39:01.230 --> 00:39:04.980
So something like
1 over 1 plus s.

00:39:07.670 --> 00:39:09.389
So how do we get there?

00:39:09.389 --> 00:39:11.430
So this is the slide--
guys, if you pay attention

00:39:11.430 --> 00:39:13.790
to one slide this
whole talk, this

00:39:13.790 --> 00:39:15.675
is the slide to
pay attention to.

00:39:15.675 --> 00:39:17.800
This is the slide where we
map the ace, king, queen

00:39:17.800 --> 00:39:19.130
game to actual poker.

00:39:19.130 --> 00:39:22.030
So the whole idea of poker
from a game theory perspective

00:39:22.030 --> 00:39:24.380
is we're going to try
to make bluffing zero

00:39:24.380 --> 00:39:25.510
EV for our opponents.

00:39:25.510 --> 00:39:28.130
So we're going to call with
the proportion of our hands.

00:39:28.130 --> 00:39:30.960
We're going to make
bluffing zero EV.

00:39:30.960 --> 00:39:33.520
So in real poker, games
that allow raising,

00:39:33.520 --> 00:39:37.440
that means we could
potentially be raising.

00:39:37.440 --> 00:39:40.270
So we want to be
continuing 1 over 1 plus s

00:39:40.270 --> 00:39:43.680
at the time, at least
calling, pondering arrays.

00:39:43.680 --> 00:39:45.880
So what do we see?

00:39:45.880 --> 00:39:47.970
So if this is the
ace, king, queen game,

00:39:47.970 --> 00:39:50.040
we're going to map it
to real poker this way.

00:39:50.040 --> 00:39:53.340
This is like our 99th percentile
hand, the very best hand we

00:39:53.340 --> 00:39:55.260
can have in this spot, right?

00:39:55.260 --> 00:39:57.610
So on that board there were
two kings on the board,

00:39:57.610 --> 00:40:00.945
so a 99th percentile hand would
be like quad kings, right?

00:40:00.945 --> 00:40:02.620
Would be four of a kind.

00:40:02.620 --> 00:40:05.086
This is our worst hand, the
very worst hand we could have

00:40:05.086 --> 00:40:06.211
in this spot, I don't know.

00:40:06.211 --> 00:40:09.170
A four deuce.

00:40:09.170 --> 00:40:11.460
And so what we're
going to do-- so again,

00:40:11.460 --> 00:40:14.120
these are the hands that are
the very best at showdown,

00:40:14.120 --> 00:40:16.020
very high EV.

00:40:16.020 --> 00:40:19.540
Very worst at
showdown, very low EV.

00:40:19.540 --> 00:40:21.770
So we're going to be
calling or raising 1 over 1

00:40:21.770 --> 00:40:24.290
plus s of the time.

00:40:24.290 --> 00:40:27.830
So the question is going to be,
what is our distribution here?

00:40:27.830 --> 00:40:31.520
What is 1 over 1 plus s?

00:40:31.520 --> 00:40:33.550
Our bluffed value ratio
is going to be s over 1

00:40:33.550 --> 00:40:36.380
plus s of our
worst hands, right?

00:40:36.380 --> 00:40:39.420
These are the hands that gain
the most value by bluffing,

00:40:39.420 --> 00:40:39.920
right?

00:40:39.920 --> 00:40:43.790
The ones that are going to
do the worst at showdown.

00:40:43.790 --> 00:40:46.422
Well, not for us in this spot.

00:40:46.422 --> 00:40:47.880
I'm going to talk
about that later.

00:40:47.880 --> 00:40:50.430
Let's leave that for a second.

00:40:50.430 --> 00:40:52.960
But those are the hands that
my opponent should be bluffing,

00:40:52.960 --> 00:40:53.460
right?

00:40:56.080 --> 00:40:58.270
So if we think about
our value betting range,

00:40:58.270 --> 00:41:01.370
I might have a different
value betting range than you.

00:41:01.370 --> 00:41:04.600
And that has implications
for how we play differently

00:41:04.600 --> 00:41:05.610
than each other.

00:41:05.610 --> 00:41:07.685
So the wider my
value betting range

00:41:07.685 --> 00:41:13.050
is, the more hands I
need to be bluffing.

00:41:13.050 --> 00:41:16.840
As this region expands,
this region expands.

00:41:16.840 --> 00:41:20.770
So if my opponent is
value betting more hands,

00:41:20.770 --> 00:41:23.040
they should also
be bluffing more.

00:41:23.040 --> 00:41:26.610
So when you see some of the
very best no-limit players

00:41:26.610 --> 00:41:29.230
play these guys who are just
complete animals-- you're like,

00:41:29.230 --> 00:41:31.610
how are they
bluffing that there?

00:41:31.610 --> 00:41:33.290
How are they calling that there?

00:41:33.290 --> 00:41:37.290
How are they making
this value bet so thin?

00:41:37.290 --> 00:41:40.110
This how they do it, right?

00:41:40.110 --> 00:41:43.380
If they're value betting a lot,
they're also bluffing a lot.

00:41:43.380 --> 00:41:45.010
So you have to call them more.

00:41:45.010 --> 00:41:47.787
So the more they bluff, the
more they have to value bet.

00:41:47.787 --> 00:41:49.870
This is one of the places
where a lot of beginners

00:41:49.870 --> 00:41:54.110
just get way, way out of
whack, because bluffing is--

00:41:54.110 --> 00:41:56.660
it starts scary and
then it gets sexy,

00:41:56.660 --> 00:42:01.210
and then it gets
something in between.

00:42:01.210 --> 00:42:04.330
So the whole idea, don't
let this get out of whack.

00:42:06.990 --> 00:42:11.060
And again, the larger amount
that my opponent bets,

00:42:11.060 --> 00:42:12.520
the less frequently
I have to call.

00:42:12.520 --> 00:42:15.290
The less they bet, the more
frequently I have to call.

00:42:15.290 --> 00:42:19.720
So we're about to our own hand.

00:42:19.720 --> 00:42:22.450
Let's keep these couple
of things in mind.

00:42:22.450 --> 00:42:24.200
So again, I think
reading your own hand is

00:42:24.200 --> 00:42:25.700
the most important
skill in poker,

00:42:25.700 --> 00:42:28.330
and it's because what you do
with part of your distribution

00:42:28.330 --> 00:42:30.174
shapes what you do
with the rest of it.

00:42:30.174 --> 00:42:32.590
And so what we're about to do
is we're going to go through

00:42:32.590 --> 00:42:34.506
and we're going to sort
of make some frequency

00:42:34.506 --> 00:42:37.660
updates on each street.

00:42:37.660 --> 00:42:39.320
So we're going to
do two updates.

00:42:39.320 --> 00:42:41.942
We're going to sort of update
what our hands might be given

00:42:41.942 --> 00:42:43.150
the cards that have come out.

00:42:43.150 --> 00:42:46.030
So there's a card
removal effect, right?

00:42:46.030 --> 00:42:48.570
If an ace comes out,
I'm a lot less likely

00:42:48.570 --> 00:42:51.900
to have a pair of aces because
there are fewer combinations.

00:42:51.900 --> 00:42:55.540
And then we're also going
to account for the actions

00:42:55.540 --> 00:42:56.780
that we take.

00:42:56.780 --> 00:42:58.720
So let's just do it.

00:42:58.720 --> 00:42:59.350
It'll be clear.

00:42:59.350 --> 00:42:59.850
So, OK.

00:42:59.850 --> 00:43:01.266
So I've opened two
off the button.

00:43:01.266 --> 00:43:03.369
So here's some kind
of reasonable range

00:43:03.369 --> 00:43:04.660
for opening two off the button.

00:43:04.660 --> 00:43:06.910
So we've gotten rid of all
the hands that aren't here.

00:43:06.910 --> 00:43:09.750
This is what I might
have right now, OK?

00:43:09.750 --> 00:43:11.940
So you don't really
need to pay attention

00:43:11.940 --> 00:43:15.360
to the specifics here, but
flop comes, king, jack, eight.

00:43:15.360 --> 00:43:18.170
So now I'm going to
update for card removal.

00:43:18.170 --> 00:43:20.469
So all the hands in
orange are the frequencies

00:43:20.469 --> 00:43:21.510
that have changed, right?

00:43:21.510 --> 00:43:24.424
So eights, I could
have had six of them

00:43:24.424 --> 00:43:26.590
before the flop came, but
now that there's an eight,

00:43:26.590 --> 00:43:28.810
I only have three
of them, et cetera.

00:43:28.810 --> 00:43:31.080
So our total combination
down the bottom

00:43:31.080 --> 00:43:33.950
here has gone down, right?

00:43:33.950 --> 00:43:35.630
So now what happens?

00:43:35.630 --> 00:43:38.680
Well, they check, I bet.

00:43:38.680 --> 00:43:41.270
So what hands am I betting
here when my opponent checks?

00:43:41.270 --> 00:43:42.440
Well, I'm getting
rid of some of them.

00:43:42.440 --> 00:43:44.356
I don't know if they're
the right ones or not,

00:43:44.356 --> 00:43:49.280
but I'm getting rid of some
pairs, some bottom pairs,

00:43:49.280 --> 00:43:50.590
some gut shots.

00:43:50.590 --> 00:43:53.500
I might not be betting
those here, right?

00:43:53.500 --> 00:43:57.020
So the hands in white are all
the hands I'm still betting.

00:43:57.020 --> 00:43:59.520
And again, our frequency
is coming down.

00:43:59.520 --> 00:44:02.061
Our distribution is narrowing.

00:44:02.061 --> 00:44:02.810
So the turn comes.

00:44:02.810 --> 00:44:03.350
It's a five.

00:44:03.350 --> 00:44:07.570
We get rid of some hands
with fives in them.

00:44:07.570 --> 00:44:08.756
I bet 2/3 of the pot.

00:44:08.756 --> 00:44:11.130
So we get rid of all the hands
in this distribution where

00:44:11.130 --> 00:44:14.360
I'm not betting 2/3 of the pot.

00:44:14.360 --> 00:44:17.960
And really quickly,
when you're thinking

00:44:17.960 --> 00:44:19.530
about reading your
own hand-- and I'm

00:44:19.530 --> 00:44:21.320
going to say that the most
important time to be doing

00:44:21.320 --> 00:44:23.930
it is probably off the table--
when you're thinking about it,

00:44:23.930 --> 00:44:26.950
you should be really thinking
about every street, OK?

00:44:26.950 --> 00:44:27.450
So OK.

00:44:27.450 --> 00:44:29.870
So these are the hands
that I'm value betting.

00:44:29.870 --> 00:44:34.190
Turns out I'm value betting
about 94 combinations here.

00:44:34.190 --> 00:44:36.130
How's my proportion of bluffs?

00:44:36.130 --> 00:44:37.775
So I'm value betting
94 combinations.

00:44:37.775 --> 00:44:39.990
I have a total of 120.

00:44:39.990 --> 00:44:41.420
So that leaves what?

00:44:41.420 --> 00:44:42.970
26 hands.

00:44:42.970 --> 00:44:46.660
So how often should
I be bluffing?

00:44:46.660 --> 00:44:48.620
Or-- hm.

00:44:48.620 --> 00:44:49.330
96.

00:44:49.330 --> 00:44:52.354
I'm value betting 96
leaving 24 bluff hands.

00:44:52.354 --> 00:44:53.770
So how often should
I be bluffing?

00:44:53.770 --> 00:44:57.590
OK, so 96 times-- what's that?

00:44:57.590 --> 00:45:03.900
Is it like 1 over 1 plus s, s
over 1 plus s, s over 1 plus s?

00:45:03.900 --> 00:45:07.090
So times 0.4-- that's
the amount I'm betting,

00:45:07.090 --> 00:45:11.060
I'm betting 40% of the
pot-- times 1 plus 0.4.

00:45:11.060 --> 00:45:14.930
So 96 times 0.4.

00:45:14.930 --> 00:45:15.610
Yeah.

00:45:15.610 --> 00:45:17.980
96 times 0.4 over 1.4.

00:45:17.980 --> 00:45:18.480
All right.

00:45:18.480 --> 00:45:22.175
So I should be bluffing
about 27 hands.

00:45:22.175 --> 00:45:22.800
What do I have?

00:45:22.800 --> 00:45:24.320
16, 24?

00:45:24.320 --> 00:45:24.860
Huh.

00:45:24.860 --> 00:45:25.526
Did pretty well.

00:45:25.526 --> 00:45:26.500
I'm happy with that.

00:45:26.500 --> 00:45:27.549
Pretty good shape.

00:45:27.549 --> 00:45:28.090
I don't know.

00:45:28.090 --> 00:45:29.030
I don't know about
all the other actions.

00:45:29.030 --> 00:45:30.863
We can argue about what
I'm checking behind,

00:45:30.863 --> 00:45:33.100
but we don't want
to over think it.

00:45:33.100 --> 00:45:36.020
So generally speaking, I want to
be checking in on each street.

00:45:36.020 --> 00:45:37.020
Like, ooh.

00:45:37.020 --> 00:45:41.640
Am I balanced here in the way
that I should be balanced?

00:45:41.640 --> 00:45:46.680
So the river comes a king, and
now we have fewer combinations.

00:45:46.680 --> 00:45:49.110
So what do I do with my aces?

00:45:49.110 --> 00:45:53.115
So again, s, the bet
size, he bets $1,080,000

00:45:53.115 --> 00:45:55.960
into a $720,000 chip pot.

00:45:55.960 --> 00:45:57.670
So s is 1.5.

00:45:57.670 --> 00:46:02.350
I should be calling
1.5 over 2.5, 40%.

00:46:02.350 --> 00:46:05.440
That's 40% of hands
that beat a bluff.

00:46:05.440 --> 00:46:08.120
So what does that look like?

00:46:08.120 --> 00:46:11.122
So if we're thinking
about our calling region,

00:46:11.122 --> 00:46:13.580
what I've done here is I've
just sort of taken all my hands

00:46:13.580 --> 00:46:19.150
and I've ranked them, and so
kings represent my top 1%, king

00:46:19.150 --> 00:46:20.260
jack.

00:46:20.260 --> 00:46:21.085
Now we're at 9%.

00:46:21.085 --> 00:46:23.585
This is like the cumulative
sort of frequency distribution

00:46:23.585 --> 00:46:24.460
with my hands ranked.

00:46:28.590 --> 00:46:30.490
And I'm saying here
in this distribution,

00:46:30.490 --> 00:46:33.660
queen jack is the worst hand
that still beats a bluff.

00:46:33.660 --> 00:46:38.130
We can argue about
that, but rough guess.

00:46:38.130 --> 00:46:43.800
So first nothing in excess,
calling 1 over 1 plus

00:46:43.800 --> 00:46:46.730
s of a time.

00:46:46.730 --> 00:46:47.637
Where are we?

00:46:47.637 --> 00:46:48.890
Ooh.

00:46:48.890 --> 00:46:50.542
This is surprising to me.

00:46:50.542 --> 00:46:52.560
This felt like a
tough decision, right?

00:46:52.560 --> 00:46:55.480
So we're actually at the
62nd percentile here.

00:46:55.480 --> 00:47:01.130
So one question is, so from
this, what should I be doing?

00:47:01.130 --> 00:47:03.400
Should I be calling or folding?

00:47:03.400 --> 00:47:04.180
Folding.

00:47:04.180 --> 00:47:05.540
Yeah.

00:47:05.540 --> 00:47:07.540
So it looks like I'm
calling with my king queen.

00:47:07.540 --> 00:47:09.860
I might be folding
king 10 maybe.

00:47:09.860 --> 00:47:12.010
We can talk about
that a little bit.

00:47:12.010 --> 00:47:14.820
But aces seem like a
pretty clear fold here.

00:47:18.640 --> 00:47:22.890
Suppose my opponent thinks that
I'll fold good hands like aces,

00:47:22.890 --> 00:47:24.720
and so he's like, mm.

00:47:24.720 --> 00:47:26.865
I'm going to bluff
90% of the time here.

00:47:26.865 --> 00:47:28.250
Is he exploiting me?

00:47:34.710 --> 00:47:37.756
Seeing head shakes.

00:47:37.756 --> 00:47:39.110
He's not exploiting me, right?

00:47:39.110 --> 00:47:39.800
So why not?

00:47:43.252 --> 00:47:43.752
Yeah?

00:47:43.752 --> 00:47:46.716
AUDIENCE: Because
you win on his bluff.

00:47:46.716 --> 00:47:49.383
Such a big question at the
time, it makes up for the pot

00:47:49.383 --> 00:47:50.174
that you're losing.

00:47:50.174 --> 00:47:51.090
MATT HAWRILENKO: Yeah.

00:47:51.090 --> 00:47:54.126
My distribution is really
strong here, right?

00:47:54.126 --> 00:48:00.479
Of the hands that beat a bluff,
like half of them have trips.

00:48:00.479 --> 00:48:01.270
That's crazy to me.

00:48:04.190 --> 00:48:04.800
So no.

00:48:04.800 --> 00:48:06.591
So he's not exploiting
me, but we will talk

00:48:06.591 --> 00:48:09.680
about exploitation in a minute.

00:48:09.680 --> 00:48:11.830
So we solved it, right?

00:48:11.830 --> 00:48:12.860
So we fold aces.

00:48:12.860 --> 00:48:15.534
We even fold king 10.

00:48:15.534 --> 00:48:16.950
So one question
is, do we actually

00:48:16.950 --> 00:48:18.220
want to have a
distribution in the spot

00:48:18.220 --> 00:48:19.386
where we have to fold trips?

00:48:21.860 --> 00:48:27.160
And if that feels kind of
bad, well, two possibilities.

00:48:27.160 --> 00:48:29.460
One it just feels
bad, or two, it

00:48:29.460 --> 00:48:32.570
might mean we sort of
screwed up on the way here.

00:48:32.570 --> 00:48:34.050
It's not the worst.

00:48:34.050 --> 00:48:36.000
It's not a four flush board.

00:48:36.000 --> 00:48:37.860
There are no straights
on it, right?

00:48:37.860 --> 00:48:40.050
Some folding trips here,
that feels kind of bad.

00:48:40.050 --> 00:48:41.170
So what might that mean?

00:48:41.170 --> 00:48:44.080
A couple of things, right?

00:48:44.080 --> 00:48:48.000
So one thing we could
do is I probably

00:48:48.000 --> 00:48:50.692
want a distribution where
I don't have to fold it.

00:48:50.692 --> 00:48:52.650
And again, it depends a
little bit on bet size,

00:48:52.650 --> 00:48:54.714
but this is a pretty
reasonable rule of thumb

00:48:54.714 --> 00:48:56.130
when you're thinking
about shaping

00:48:56.130 --> 00:48:57.630
your play off the table.

00:48:57.630 --> 00:49:00.490
So one thing that we can
do, we can add some hands

00:49:00.490 --> 00:49:01.370
in from earlier.

00:49:01.370 --> 00:49:03.100
Ooh, maybe I should have
played the turn a little bit

00:49:03.100 --> 00:49:03.600
differently.

00:49:03.600 --> 00:49:05.120
Maybe I should have
played preflop,

00:49:05.120 --> 00:49:07.203
maybe I should have played
some more hands, right?

00:49:07.203 --> 00:49:09.290
So we can start to
expand our range.

00:49:09.290 --> 00:49:10.190
Right?

00:49:10.190 --> 00:49:12.440
Now we're calling with
most all of our kings.

00:49:14.944 --> 00:49:16.860
We can actually construct
a distribution where

00:49:16.860 --> 00:49:19.690
we have to call it aces, right?

00:49:19.690 --> 00:49:22.430
If we're playing way
more hands and if we're

00:49:22.430 --> 00:49:26.790
value betting way more hands
throughout, all of a sudden

00:49:26.790 --> 00:49:29.240
our distribution
is wider, right?

00:49:29.240 --> 00:49:32.780
And now I should be in the
same spot with the same hand

00:49:32.780 --> 00:49:36.320
and do something differently
because my strategy to get here

00:49:36.320 --> 00:49:37.830
was different.

00:49:37.830 --> 00:49:39.390
And again, that's the same idea.

00:49:39.390 --> 00:49:41.630
You see these
really good players

00:49:41.630 --> 00:49:45.980
making these crazily thin
value bets, and this is why.

00:49:45.980 --> 00:49:50.339
They're bluffing a lot
so they value bet a lot.

00:49:50.339 --> 00:49:52.130
The main idea that I
want you to take away,

00:49:52.130 --> 00:49:54.020
if we sort of think
of a principle,

00:49:54.020 --> 00:49:56.640
if I find myself on the river
and I have more medium strength

00:49:56.640 --> 00:49:59.810
hands, I have to call more
with medium strength hands,

00:49:59.810 --> 00:50:02.890
otherwise I can get exploited.

00:50:02.890 --> 00:50:06.399
Suppose we were deeper and
our opponent bets into us,

00:50:06.399 --> 00:50:07.690
and we have a bunch more chips.

00:50:07.690 --> 00:50:11.080
Which hands should we be bluff
raising from this distribution?

00:50:11.080 --> 00:50:14.920
So most the time, if I'm
just straight bluffing,

00:50:14.920 --> 00:50:18.170
I want to be bluffing with the
very bottom of my distribution

00:50:18.170 --> 00:50:20.110
because that's the
part the gains the most

00:50:20.110 --> 00:50:21.470
when my opponent folds.

00:50:21.470 --> 00:50:25.740
If I'm bluff raising,
it's a little different.

00:50:25.740 --> 00:50:28.450
If I'm bluff raising, I
want to think about, OK,

00:50:28.450 --> 00:50:30.980
so what is the set of
hands I would fold?

00:50:30.980 --> 00:50:35.270
What are the very best hands
in that set that I would fold?

00:50:35.270 --> 00:50:37.230
And I may as well
choose those, right?

00:50:37.230 --> 00:50:40.610
Because it seems like those
hands have more value, so

00:50:40.610 --> 00:50:42.190
if I'm going to
fold them anyway,

00:50:42.190 --> 00:50:45.320
it's a dominated
strategy to bluff raise

00:50:45.320 --> 00:50:48.917
with hands that are
weaker than that, right?

00:50:48.917 --> 00:50:50.750
So if I'm bluff raising,
it should be really

00:50:50.750 --> 00:50:53.050
with the very best hands
that I would otherwise

00:50:53.050 --> 00:50:55.345
fold, not with the very
bottom of my distribution.

00:51:00.590 --> 00:51:02.060
So what do we see here?

00:51:02.060 --> 00:51:05.034
So first off, we want to check
for balance on all streets.

00:51:05.034 --> 00:51:06.450
That's the big
thing we take away.

00:51:06.450 --> 00:51:11.050
We can argue about little
bits of distributions,

00:51:11.050 --> 00:51:14.280
but really, we're not solving
to the second decimal point

00:51:14.280 --> 00:51:16.060
here, right?

00:51:16.060 --> 00:51:18.800
Second, we can look
at this board all day.

00:51:18.800 --> 00:51:23.419
We can look at this king jack
eight five king board all day.

00:51:23.419 --> 00:51:25.585
Unfortunately, that's not
going to be the board that

00:51:25.585 --> 00:51:28.940
comes every time, and if we
just look at this board all day,

00:51:28.940 --> 00:51:31.720
we're going to start to overfit
our strategy a little bit

00:51:31.720 --> 00:51:33.040
to this board.

00:51:33.040 --> 00:51:40.550
So what I suggest you focus on
is fixing the glaring errors,

00:51:40.550 --> 00:51:43.400
and there will be
glaring errors, places

00:51:43.400 --> 00:51:45.730
where your distribution
is way, way imbalanced.

00:51:48.529 --> 00:51:50.445
I still find glaring
errors when I play poker,

00:51:50.445 --> 00:51:54.540
and I've been trying to do
this for a little while now.

00:51:54.540 --> 00:51:57.370
Another thought which is not
quite so obvious from what I've

00:51:57.370 --> 00:51:59.950
said so far is, you don't
want to needlessly bifurcate

00:51:59.950 --> 00:52:00.710
your distribution.

00:52:00.710 --> 00:52:02.340
How do you bifurcate
your distribution?

00:52:02.340 --> 00:52:07.790
Well, suppose preflop,
I raise some amount

00:52:07.790 --> 00:52:10.750
with some set of hands
and in a different amount

00:52:10.750 --> 00:52:12.970
with a different set of hands.

00:52:12.970 --> 00:52:15.370
Ooh, right?

00:52:15.370 --> 00:52:18.730
All of a sudden I started off
here with 310 combinations,

00:52:18.730 --> 00:52:22.326
and I got down to, I don't
know, 60 or something.

00:52:22.326 --> 00:52:24.700
But now all of a sudden, I'm
starting off with half that.

00:52:24.700 --> 00:52:29.050
I'm starting off with 155, in
this game tree gets smaller

00:52:29.050 --> 00:52:30.720
really, really quickly.

00:52:30.720 --> 00:52:32.590
So I'm not saying don't do it.

00:52:32.590 --> 00:52:36.070
But I'm saying if you do it,
A, have a really good reason

00:52:36.070 --> 00:52:37.190
for doing it.

00:52:37.190 --> 00:52:40.060
B, be really careful.

00:52:40.060 --> 00:52:42.420
Be really careful of
betting different amounts

00:52:42.420 --> 00:52:43.740
with different hand types.

00:52:43.740 --> 00:52:45.750
I basically don't.

00:52:45.750 --> 00:52:47.470
I will bet different
amounts based

00:52:47.470 --> 00:52:48.990
on the texture of the board.

00:52:48.990 --> 00:52:50.470
So if boards are
more [? drawy ?]

00:52:50.470 --> 00:52:54.590
I might tend to make my
bet sizes larger earlier.

00:52:54.590 --> 00:52:57.350
But I won't bet different
amounts with different hand

00:52:57.350 --> 00:53:01.700
types because I think it the
possible gain is so small

00:53:01.700 --> 00:53:05.881
and the possible loss is so big.

00:53:05.881 --> 00:53:08.130
And the last thing that I
found pretty cool about this

00:53:08.130 --> 00:53:10.840
was when I started
really, really spending

00:53:10.840 --> 00:53:12.490
time trying to read
my own hand, I'd

00:53:12.490 --> 00:53:14.700
start to find these
consistent situations where

00:53:14.700 --> 00:53:17.047
I would get really imbalanced.

00:53:17.047 --> 00:53:19.380
And then when you want to
start to think about moving on

00:53:19.380 --> 00:53:21.335
to exploitive play
after you have paid off

00:53:21.335 --> 00:53:24.220
a house with your
poker winnings,

00:53:24.220 --> 00:53:28.400
you want to maybe identify those
spots in your own opponents,

00:53:28.400 --> 00:53:28.910
right?

00:53:28.910 --> 00:53:30.368
If you're getting
imbalanced there,

00:53:30.368 --> 00:53:35.100
other people probably are
too, so be ready for that.

00:53:35.100 --> 00:53:40.660
So what do I want to say here?

00:53:40.660 --> 00:53:43.950
So I guess we've ignored a
few assumptions of this model,

00:53:43.950 --> 00:53:45.180
right?

00:53:45.180 --> 00:53:47.920
The biggest assumption in a toy
game like the ace king queen

00:53:47.920 --> 00:53:50.340
game or the Coin Flip
Clairvoyance game

00:53:50.340 --> 00:53:54.450
is that distributions
are symmetric.

00:53:54.450 --> 00:53:58.970
If I'm applying the ace
king queen game to my river

00:53:58.970 --> 00:54:02.220
situation here, I'm
implicitly assuming

00:54:02.220 --> 00:54:04.780
that my opponents
and I essentially

00:54:04.780 --> 00:54:08.450
have an equivalent distribution
of aces, kings, and queens.

00:54:08.450 --> 00:54:11.010
That's not always true in poker.

00:54:11.010 --> 00:54:18.160
So generally I'd say that this
sort this mapping is actually

00:54:18.160 --> 00:54:21.220
fairly robust, but
at the same time,

00:54:21.220 --> 00:54:22.949
you have to be
aware of situations

00:54:22.949 --> 00:54:24.990
where your distributions
aren't symmetric, right?

00:54:24.990 --> 00:54:27.980
Like if you raise under the
gun at a ten handed table

00:54:27.980 --> 00:54:30.450
and the big blind
calls, your distribution

00:54:30.450 --> 00:54:31.600
is way, way stronger.

00:54:31.600 --> 00:54:32.520
It's not symmetric.

00:54:32.520 --> 00:54:34.730
So you're going to have a
different set of actions.

00:54:34.730 --> 00:54:38.360
However, as you progress
throughout the hand,

00:54:38.360 --> 00:54:42.700
distributions sort of tend to
become more and more symmetric.

00:54:42.700 --> 00:54:46.550
So generally speaking, it's
a model with some assumptions

00:54:46.550 --> 00:54:49.140
which are meh.

00:54:49.140 --> 00:54:51.650
But in my experience,
it holds pretty well.

00:54:55.005 --> 00:54:56.880
So another thing I
haven't talked about today

00:54:56.880 --> 00:54:59.850
is I haven't told you which
hands to value bet, right?

00:54:59.850 --> 00:55:01.800
So I've told you the
calling proportion,

00:55:01.800 --> 00:55:02.930
bluffing proportion.

00:55:02.930 --> 00:55:05.750
I haven't said anything about,
how do you choose what to bet?

00:55:05.750 --> 00:55:08.931
And there are game theory
games that can kind of give you

00:55:08.931 --> 00:55:09.680
insight into that.

00:55:09.680 --> 00:55:12.280
You can get insight into that
from Bill and Jared's book.

00:55:12.280 --> 00:55:18.890
But I think that
what's most important

00:55:18.890 --> 00:55:21.410
here is reading your own
hand is the thing that I

00:55:21.410 --> 00:55:24.480
think can integrate
into everybody's game.

00:55:24.480 --> 00:55:27.720
Whatever hands you currently
value bet, it can work for you.

00:55:27.720 --> 00:55:28.220
Right?

00:55:28.220 --> 00:55:31.185
And it can sort of work equally
well for the right player

00:55:31.185 --> 00:55:33.440
as it can for the
very loose player.

00:55:33.440 --> 00:55:36.290
So the reason I've decided
to talk about the reading

00:55:36.290 --> 00:55:38.910
your own hand approach is this
is a very flexible approach

00:55:38.910 --> 00:55:41.380
that can work for a lot of
people in a lot of situations.

00:55:44.164 --> 00:55:45.830
And, you know, as you
get better and you

00:55:45.830 --> 00:55:48.650
start to play more and more
hands-- or fewer and fewer,

00:55:48.650 --> 00:55:50.400
depending where you
fall on the spectrum--

00:55:50.400 --> 00:55:52.320
but generally as you start to
play more and more hands, what

00:55:52.320 --> 00:55:54.860
you're doing here, thinking
from sort of like this game

00:55:54.860 --> 00:55:57.790
theoretic perspective
is you're moving

00:55:57.790 --> 00:56:02.130
towards hands that are closer to
threshold hands, closer to zero

00:56:02.130 --> 00:56:02.860
easy.

00:56:02.860 --> 00:56:07.350
So adding in the nth hand
has a much smaller effect

00:56:07.350 --> 00:56:11.100
than adding in that,
like, fifth hand, right?

00:56:11.100 --> 00:56:14.080
Or if you think about all the
good hands in poker, right?

00:56:14.080 --> 00:56:17.027
So you're always going
to play your jacks,

00:56:17.027 --> 00:56:18.860
you're always going to
play your 10s, right?

00:56:18.860 --> 00:56:24.350
Adding in that nine six suited
has a much smaller impact

00:56:24.350 --> 00:56:28.510
on your EV than adding in the
10s, the nines, the eights.

00:56:31.040 --> 00:56:32.876
Yeah, that's all I
want to say there.

00:56:32.876 --> 00:56:35.000
So in the last few minutes
I'm going to talk really

00:56:35.000 --> 00:56:36.680
briefly about exploitive play.

00:56:36.680 --> 00:56:38.700
So exploitive play.

00:56:38.700 --> 00:56:41.670
So if we're looking at the top,
our best hand's at showdown,

00:56:41.670 --> 00:56:42.880
our worst hand's at showdown.

00:56:42.880 --> 00:56:45.171
And this is about where our
value betting threshold is.

00:56:47.710 --> 00:56:50.650
The way to exploit from a
game theoretic perspective

00:56:50.650 --> 00:56:53.780
isn't to say, ah, I
know he's bluffing!

00:56:53.780 --> 00:56:55.300
I call a blind!

00:56:55.300 --> 00:56:58.800
Although apparently that can
work pretty well sometimes.

00:56:58.800 --> 00:57:01.370
The way you do it is you
expand to the margins.

00:57:01.370 --> 00:57:05.210
So if normally I'm value
betting here and my opponent--

00:57:05.210 --> 00:57:08.180
or normally I'm calling
here and my opponent bluffs

00:57:08.180 --> 00:57:10.480
way too much, I might expand
to the marginal calls.

00:57:10.480 --> 00:57:12.516
That's one thing I might do.

00:57:12.516 --> 00:57:13.890
If they don't
value bet enough, I

00:57:13.890 --> 00:57:16.910
might contract the
marginal calls, right?

00:57:16.910 --> 00:57:21.190
But again, the ones that
are very close to zero EV,

00:57:21.190 --> 00:57:23.880
I'm never making
the big fold here.

00:57:23.880 --> 00:57:25.510
If they fold too
much, I might expand

00:57:25.510 --> 00:57:26.968
to the marginal
bluffs a little bit

00:57:26.968 --> 00:57:31.100
and contract the marginal
bets a little bit.

00:57:31.100 --> 00:57:34.100
If they call too much, I'm going
to expand the marginal bets

00:57:34.100 --> 00:57:35.870
and contract the
marginal bluffs.

00:57:35.870 --> 00:57:38.370
So what that means
is you get away

00:57:38.370 --> 00:57:45.060
from, I think he's bluffing 80%
of the time in this situation,

00:57:45.060 --> 00:57:46.920
to-- in this exact
situation, and I'm

00:57:46.920 --> 00:57:50.590
going to catch him-- to, hm.

00:57:50.590 --> 00:57:52.650
This person seems
a little bluffy,

00:57:52.650 --> 00:57:54.600
so I'm going to
shape my distribution

00:57:54.600 --> 00:57:56.450
just a little bit around that.

00:57:59.380 --> 00:57:59.880
OK.

00:58:03.800 --> 00:58:06.260
So the other idea
about exploitive play

00:58:06.260 --> 00:58:08.940
is as your read grows stronger,
as you have more confidence,

00:58:08.940 --> 00:58:10.680
you can expand to
the margins, right?

00:58:10.680 --> 00:58:13.720
So a little read, you
move the margins a little.

00:58:13.720 --> 00:58:17.460
Big read, you move
the margins more.

00:58:17.460 --> 00:58:21.520
For example, well, OK.

00:58:21.520 --> 00:58:26.360
So with our example hand, we
have a margin right about here,

00:58:26.360 --> 00:58:28.220
but we can start
to move it wider

00:58:28.220 --> 00:58:30.830
as our read grow stronger that
our opponent might be bluffing.

00:58:30.830 --> 00:58:32.990
But again, we don't
want to move it

00:58:32.990 --> 00:58:36.170
wider with just the hand that I
happen to be holding right now.

00:58:36.170 --> 00:58:37.974
We need to be thinking,
how strong is it,

00:58:37.974 --> 00:58:39.640
and how wide am I
comfortable moving it?

00:58:42.944 --> 00:58:45.079
Yeah.

00:58:45.079 --> 00:58:46.620
So if we're thinking
about exploiting

00:58:46.620 --> 00:58:48.703
with my sort of, like,
shamefully exploitive hand,

00:58:48.703 --> 00:58:49.400
what did I do?

00:58:49.400 --> 00:58:52.335
Well, it wasn't quite a 1
over 1 plus s situation.

00:58:52.335 --> 00:58:53.710
We're in this
situation where I'm

00:58:53.710 --> 00:58:58.190
going to be shoving in the top
hands and folding the worst.

00:58:58.190 --> 00:59:04.410
So if my margin was here, if
my margin was, I don't know,

00:59:04.410 --> 00:59:06.410
eight six suited or
something, maybe I'd

00:59:06.410 --> 00:59:08.340
go, like, eight five suited.

00:59:08.340 --> 00:59:12.330
Maybe I'd go king deuce off
suit, something like that.

00:59:12.330 --> 00:59:14.560
Instead what I did
was I went all the way

00:59:14.560 --> 00:59:16.730
to the very bottom
of my distribution

00:59:16.730 --> 00:59:18.720
and got suitably punished.

00:59:22.050 --> 00:59:22.550
Yeah.

00:59:22.550 --> 00:59:26.760
So however confident you
think you are in your read,

00:59:26.760 --> 00:59:29.351
you are probably
overestimating it.

00:59:29.351 --> 00:59:29.850
So, yeah.

00:59:29.850 --> 00:59:31.940
That's where I was.

00:59:31.940 --> 00:59:34.720
One other thought
here before I wrap up,

00:59:34.720 --> 00:59:37.510
and that's this idea of
advanced exploitive play,

00:59:37.510 --> 00:59:39.580
and this is fun.

00:59:39.580 --> 00:59:41.990
So we think about
expanding the margins.

00:59:41.990 --> 00:59:44.320
We think, oh, in this
spot he's bluffing.

00:59:44.320 --> 00:59:46.220
I should call more.

00:59:46.220 --> 00:59:50.140
But now if you start to make
slightly more subtle reads,

00:59:50.140 --> 00:59:55.464
this is an opponent who bluffs
a little too much on the river,

00:59:55.464 --> 00:59:57.255
I could punish him on
the river by calling,

00:59:57.255 --> 00:59:59.546
or this is an opponent who
folds too much on the river.

00:59:59.546 --> 01:00:01.490
I could punish them,
or I could punish her

01:00:01.490 --> 01:00:03.067
by betting the river.

01:00:03.067 --> 01:00:04.650
But they're going
to get that feedback

01:00:04.650 --> 01:00:07.067
pretty soon if you're just
starting to hammer every river.

01:00:07.067 --> 01:00:08.524
Another thing you
can do is you can

01:00:08.524 --> 01:00:10.335
make the pot a little
bit bigger, right?

01:00:10.335 --> 01:00:11.460
So don't forget about this.

01:00:11.460 --> 01:00:12.750
This is the pot.

01:00:12.750 --> 01:00:15.810
If you start making this
bigger on earlier streets,

01:00:15.810 --> 01:00:17.580
maybe against this
particular player,

01:00:17.580 --> 01:00:19.830
I raise a little
bit more preflop.

01:00:19.830 --> 01:00:22.570
I bet a little bit
more on the flop.

01:00:22.570 --> 01:00:23.070
Right?

01:00:23.070 --> 01:00:25.940
The pot's bigger on the turn,
so now I'm betting on the turn.

01:00:25.940 --> 01:00:28.910
And now with the river I
take all my normal actions

01:00:28.910 --> 01:00:31.790
except I know that I'm
winning a few too many pots

01:00:31.790 --> 01:00:33.470
because they're
folding too much,

01:00:33.470 --> 01:00:35.800
and now the few too many
parts that I'm winning

01:00:35.800 --> 01:00:37.906
are proportionally larger.

01:00:37.906 --> 01:00:42.110
So this is this idea of
we can exploit downstream.

01:00:42.110 --> 01:00:44.232
We can anticipate
where they're weak.

01:00:44.232 --> 01:00:47.660
We cannot change our play in
that spot and tip them off,

01:00:47.660 --> 01:00:49.810
but we can change it earlier.

01:00:49.810 --> 01:00:54.470
So to wrap up, so you
want to know yourself.

01:00:54.470 --> 01:00:56.370
You want know your
own hand, right?

01:00:56.370 --> 01:00:57.695
That's the first key.

01:00:57.695 --> 01:01:00.680
The second key is
to keep it balanced,

01:01:00.680 --> 01:01:02.390
and you want to
exploit the margins.

01:01:02.390 --> 01:01:03.960
And so as we think
about Cepheus,

01:01:03.960 --> 01:01:05.760
as we think about
these algorithms coming

01:01:05.760 --> 01:01:09.834
and exploiting players, so even
in 20 years, computing power--

01:01:09.834 --> 01:01:10.500
oh, what's that?

01:01:10.500 --> 01:01:12.680
Like 1,000 times
greater or something?

01:01:12.680 --> 01:01:14.160
Is that right?

01:01:14.160 --> 01:01:14.755
Wrong?

01:01:14.755 --> 01:01:15.880
AUDIENCE: Sounds all right.

01:01:15.880 --> 01:01:16.610
MATT HAWRILENKO: Depending.

01:01:16.610 --> 01:01:17.820
Depending if you believe
Moore's law, I guess.

01:01:17.820 --> 01:01:18.430
OK.

01:01:18.430 --> 01:01:23.430
So even then, even when more
poker games are tractable,

01:01:23.430 --> 01:01:26.670
you're going to need toy
games to draw insights

01:01:26.670 --> 01:01:28.450
on what's going on.

01:01:28.450 --> 01:01:31.160
As these strategies come out
of the black boxes, if you

01:01:31.160 --> 01:01:33.990
hope to grasp them, if you
hope to hold onto them,

01:01:33.990 --> 01:01:36.940
you need these kind
of insights to be

01:01:36.940 --> 01:01:40.680
able to have some scaffolding
to start to put them on.

01:01:40.680 --> 01:01:45.020
So I think I'm done, but if I
were to summarize this talk one

01:01:45.020 --> 01:01:46.820
way, don't be this guy.

01:01:46.820 --> 01:01:48.630
Be this guy.

01:01:48.630 --> 01:01:51.460
OK, we're done.

01:01:51.460 --> 01:01:52.115
Any questions?

01:01:55.930 --> 01:01:56.942
Ooh, wait.

01:01:56.942 --> 01:01:59.400
AUDIENCE: So if you're doing,
like, Nash equilibrium, well,

01:01:59.400 --> 01:02:01.135
that guarantees
you a positive EV.

01:02:01.135 --> 01:02:03.173
But [INAUDIBLE]
that's always enough.

01:02:03.173 --> 01:02:05.770
Like, if you're in tournament
having a positive EV,

01:02:05.770 --> 01:02:08.210
your chips will grow a
little but [? maybe blinds ?]

01:02:08.210 --> 01:02:09.910
are growing faster.

01:02:09.910 --> 01:02:12.139
Is that an [INAUDIBLE]
Nash equilibrium strategy,

01:02:12.139 --> 01:02:13.680
or do you have to
deviate if you want

01:02:13.680 --> 01:02:15.110
a higher variance [INAUDIBLE]?

01:02:15.110 --> 01:02:16.776
MATT HAWRILENKO: Yeah,
so good question.

01:02:16.776 --> 01:02:20.840
So the question-- and tell me--
let me make sure we answer it--

01:02:20.840 --> 01:02:23.590
the question is, OK, well, you
can try to play an equilibrium

01:02:23.590 --> 01:02:25.694
strategy and that's fine.

01:02:25.694 --> 01:02:27.610
But if you're exploiting,
you're winning more.

01:02:27.610 --> 01:02:29.235
And so if you're
playing in tournaments

01:02:29.235 --> 01:02:31.510
or these situations,
you really need

01:02:31.510 --> 01:02:33.240
to exploit to win more money.

01:02:33.240 --> 01:02:35.250
Is that kind of-- yeah?

01:02:35.250 --> 01:02:36.230
So yeah.

01:02:36.230 --> 01:02:38.440
That idea has been
around for a long time,

01:02:38.440 --> 01:02:42.610
and I think that
that idea is driven

01:02:42.610 --> 01:02:48.010
by people not really knowing
just how strong game theory

01:02:48.010 --> 01:02:51.170
strategies might be, and just
how strong sort of like, OK.

01:02:51.170 --> 01:02:53.237
So what is a Nash
equilibria here?

01:02:53.237 --> 01:02:54.820
When you're playing
a Nash equilibria,

01:02:54.820 --> 01:02:57.403
the idea is that your opponents
are going to impale themselves

01:02:57.403 --> 01:02:59.680
on their own
mistakes, and you're

01:02:59.680 --> 01:03:02.130
trying to make as few
mistakes as possible.

01:03:02.130 --> 01:03:06.270
And how big is that, right?

01:03:06.270 --> 01:03:08.220
It's kind of an
empirical question.

01:03:08.220 --> 01:03:13.392
In my experience, it
can be a lot bigger

01:03:13.392 --> 01:03:14.350
than most people think.

01:03:14.350 --> 01:03:15.766
Because when you
start exploiting,

01:03:15.766 --> 01:03:17.480
you start making mistakes.

01:03:17.480 --> 01:03:20.180
And you start sort of
getting impaled a little bit

01:03:20.180 --> 01:03:21.920
on your own mistakes too.

01:03:21.920 --> 01:03:28.150
And my guess is for most
of us, for 99% of us,

01:03:28.150 --> 01:03:30.840
we're going to make a lot more
mistakes than we think we do.

01:03:30.840 --> 01:03:32.510
So we think we're exploiting.

01:03:32.510 --> 01:03:34.760
At the same time we're
getting exploited.

01:03:34.760 --> 01:03:41.601
So actually, thinking about
that is an empirical question,

01:03:41.601 --> 01:03:42.100
right?

01:03:42.100 --> 01:03:47.030
Like, how do human players
do right now against Cepheus?

01:03:47.030 --> 01:03:49.340
My guess is not very well.

01:03:49.340 --> 01:03:50.840
In fact, my guess
is that they're

01:03:50.840 --> 01:03:55.820
losing more than
the top players,

01:03:55.820 --> 01:03:57.450
the top human
players are winning

01:03:57.450 --> 01:03:58.960
from other human players.

01:03:58.960 --> 01:04:02.450
So that was certainly kind of
my experience a few years ago.

01:04:02.450 --> 01:04:06.354
So it's one of these things
where you'd see it, right?

01:04:06.354 --> 01:04:07.520
Like I think-- I don't know.

01:04:07.520 --> 01:04:10.020
I feel like I saw some stat
where Cepheus was beating

01:04:10.020 --> 01:04:14.020
really good professionals
for four bets per hundred

01:04:14.020 --> 01:04:16.010
hands, which is a lot.

01:04:16.010 --> 01:04:17.230
Which is a lot.

01:04:17.230 --> 01:04:20.720
Like, most top players are
winning less than half of that.

01:04:20.720 --> 01:04:22.372
So Nash equilibria, yeah.

01:04:22.372 --> 01:04:23.080
Pretty darn good.

01:04:23.080 --> 01:04:25.804
Maybe one day when human
players are better,

01:04:25.804 --> 01:04:27.220
you need to start
exploiting more.

01:04:27.220 --> 01:04:29.730
I don't think we're
anywhere close.

01:04:29.730 --> 01:04:31.696
Other questions?

01:04:31.696 --> 01:04:34.301
Anything else?

01:04:34.301 --> 01:04:35.550
All right, I guess we're done.

01:04:35.550 --> 01:04:36.049
Thanks guys.

01:04:36.049 --> 01:04:38.590
I'll stick around for a little
bit if anyone wants to chat.

01:04:38.590 --> 01:04:41.640
[APPLAUSE]