1
00:00:01,640 --> 00:00:04,040
The following content is
provided under a Creative

2
00:00:04,040 --> 00:00:05,580
Commons license.

3
00:00:05,580 --> 00:00:07,880
Your support will help
MIT OpenCourseWare

4
00:00:07,880 --> 00:00:12,270
continue to offer high quality
educational resources for free.

5
00:00:12,270 --> 00:00:14,870
To make a donation or
view additional materials

6
00:00:14,870 --> 00:00:18,830
from hundreds of MIT courses,
visit MIT OpenCourseWare

7
00:00:18,830 --> 00:00:20,000
at ocw.mit.edu.

8
00:00:22,525 --> 00:00:24,150
RUSS TEDRAKE: I've
been getting to play

9
00:00:24,150 --> 00:00:26,040
with this robot for
a few years now--

10
00:00:26,040 --> 00:00:29,490
three years of my life
basically devoted to that robot.

11
00:00:29,490 --> 00:00:34,500
It was one of the most exciting,
technically challenging,

12
00:00:34,500 --> 00:00:39,390
exhausting, stressful, but
ultimately fulfilling things

13
00:00:39,390 --> 00:00:41,020
I've ever done.

14
00:00:41,020 --> 00:00:44,520
We got to basically take this
robot, make it drive a car,

15
00:00:44,520 --> 00:00:45,900
get out of the car--

16
00:00:45,900 --> 00:00:51,135
that was tough-- open the door,
turn valves, pick up a drill,

17
00:00:51,135 --> 00:00:53,050
cut a hole out of the wall.

18
00:00:53,050 --> 00:00:54,660
Notice there's no
safety hardness.

19
00:00:54,660 --> 00:00:56,730
It's battery autonomous.

20
00:00:56,730 --> 00:00:59,910
It has a walk over
some rough terrain,

21
00:00:59,910 --> 00:01:02,010
climbed some stairs at the end.

22
00:01:02,010 --> 00:01:06,090
It had to do this in
front of an audience.

23
00:01:06,090 --> 00:01:07,920
Basically, we got two tries.

24
00:01:07,920 --> 00:01:11,640
And if your robot
breaks, it breaks, right?

25
00:01:11,640 --> 00:01:14,820
And there was a $2
million prize at the end.

26
00:01:14,820 --> 00:01:17,340
We wanted to do it not
for the $2 million prize,

27
00:01:17,340 --> 00:01:18,750
but for the technical challenge.

28
00:01:18,750 --> 00:01:22,710
And myself and a group
of students, just

29
00:01:22,710 --> 00:01:24,872
like I said, absolutely
devoted our lives to this.

30
00:01:24,872 --> 00:01:26,580
We spent all of our
waking hours on this.

31
00:01:26,580 --> 00:01:29,670
We worked incredibly,
incredibly hard.

32
00:01:29,670 --> 00:01:32,350
So just to give you a
little bit of context,

33
00:01:32,350 --> 00:01:36,180
DARPA, our national
defense funding agency,

34
00:01:36,180 --> 00:01:37,599
has gotten excited
about the idea

35
00:01:37,599 --> 00:01:39,390
of these grant challenges,
which get people

36
00:01:39,390 --> 00:01:41,030
to work really, really hard.

37
00:01:41,030 --> 00:01:43,940
The self-driving cars
were the first one.

38
00:01:43,940 --> 00:01:46,500
MIT had a very successful
team in the Urban Challenge

39
00:01:46,500 --> 00:01:48,240
led by John.

40
00:01:48,240 --> 00:01:52,590
And then it's unquestionably
had transition impact

41
00:01:52,590 --> 00:01:56,070
into the world via Google,
Uber, Apple, and John

42
00:01:56,070 --> 00:01:58,260
will tell you all about it.

43
00:01:58,260 --> 00:02:00,780
I think in 2012, DARPA was
scratching their heads, saying,

44
00:02:00,780 --> 00:02:02,196
people haven't
worked hard enough.

45
00:02:02,196 --> 00:02:05,100
And what's the new
challenge going to be?

46
00:02:05,100 --> 00:02:07,080
And right around
that time, there

47
00:02:07,080 --> 00:02:11,430
was a disaster that maybe
helped focus their attention

48
00:02:11,430 --> 00:02:13,390
towards disaster response.

49
00:02:13,390 --> 00:02:15,750
So ultimately, it
was October 2012

50
00:02:15,750 --> 00:02:17,940
that everything started
with this kickoff

51
00:02:17,940 --> 00:02:20,610
for the DARPA
Robotics Challenge.

52
00:02:20,610 --> 00:02:24,690
The official challenge was
cast in the light of disaster

53
00:02:24,690 --> 00:02:30,030
response using the scenario
of the nuclear disaster

54
00:02:30,030 --> 00:02:31,680
as a backdrop.

55
00:02:31,680 --> 00:02:33,960
But I think really
their goal was

56
00:02:33,960 --> 00:02:36,150
to evaluate and advance
the state of the art

57
00:02:36,150 --> 00:02:38,010
in mobile manipulation.

58
00:02:38,010 --> 00:02:40,050
So if I'm the funding
agency, what I think

59
00:02:40,050 --> 00:02:42,780
is that you see
hardware coming out

60
00:02:42,780 --> 00:02:44,680
of industry that is fantastic.

61
00:02:44,680 --> 00:02:48,090
So Boston Dynamics was
building these walking robots

62
00:02:48,090 --> 00:02:48,866
and the like.

63
00:02:48,866 --> 00:02:50,490
This one is the one
we've been playing,

64
00:02:50,490 --> 00:02:54,226
Atlas, built by Boston
Dynamics, which is now Google.

65
00:02:54,226 --> 00:02:55,510
AUDIENCE: Alphabet.

66
00:02:55,510 --> 00:02:57,500
RUSS TEDRAKE: Alphabet, yeah.

67
00:02:57,500 --> 00:02:59,789
And then I think from
the research labs,

68
00:02:59,789 --> 00:03:01,830
we've been seeing really
sophisticated algorithms

69
00:03:01,830 --> 00:03:04,860
coming out but on
relatively modest hardware.

70
00:03:04,860 --> 00:03:07,170
And I think it was time
for a mash up, right?

71
00:03:07,170 --> 00:03:10,560
So they were very
interesting in the way

72
00:03:10,560 --> 00:03:11,760
they set up the competition.

73
00:03:11,760 --> 00:03:15,454
It wasn't about making it a
completely autonomous robot.

74
00:03:15,454 --> 00:03:16,620
There was there was a twist.

75
00:03:16,620 --> 00:03:18,600
You could have a human
operator, but they

76
00:03:18,600 --> 00:03:20,122
wanted to encourage autonomy.

77
00:03:20,122 --> 00:03:22,080
So what they did is they
had a degraded network

78
00:03:22,080 --> 00:03:25,409
link between the human and
the robot and some reward

79
00:03:25,409 --> 00:03:27,450
for going a little bit
faster than the other guy.

80
00:03:27,450 --> 00:03:29,659
So the idea would be that
if you had to stop and work

81
00:03:29,659 --> 00:03:31,408
over the degraded
network link and control

82
00:03:31,408 --> 00:03:33,120
every joint of your
robot, then you're

83
00:03:33,120 --> 00:03:35,036
going to be slower than
the guy whose robot is

84
00:03:35,036 --> 00:03:37,260
making the decisions by itself.

85
00:03:37,260 --> 00:03:39,240
That didn't play out
as much as we expected,

86
00:03:39,240 --> 00:03:41,040
but that was the setup.

87
00:03:41,040 --> 00:03:42,930
That set up a
spectrum where people

88
00:03:42,930 --> 00:03:45,720
could do full
teleoperation, meaning

89
00:03:45,720 --> 00:03:48,660
joystick control of each of
the joints if they wanted to.

90
00:03:48,660 --> 00:03:51,360
And maybe the goal is to
have complete autonomy,

91
00:03:51,360 --> 00:03:54,000
and you can pick your
place on the spectrum.

92
00:03:54,000 --> 00:03:57,660
So MIT, possibly to a fault,
aimed for the full autonomy

93
00:03:57,660 --> 00:03:58,590
side.

94
00:03:58,590 --> 00:04:03,480
The idea was, let's just get
a few clicks of information

95
00:04:03,480 --> 00:04:04,630
from the human.

96
00:04:04,630 --> 00:04:06,876
Let the human solve the
really, really hard problems

97
00:04:06,876 --> 00:04:08,250
that he could
solve efficiently--

98
00:04:08,250 --> 00:04:09,090
object recognition.

99
00:04:09,090 --> 00:04:11,260
Scene understanding-- we
don't have to do that,

100
00:04:11,260 --> 00:04:13,860
but a few clicks from the
human can communicate that.

101
00:04:13,860 --> 00:04:15,870
But let the robot do
all of the dynamics

102
00:04:15,870 --> 00:04:18,810
and control and
planning side of things.

103
00:04:18,810 --> 00:04:21,600
So those few clicks should see
nearly autonomous algorithms

104
00:04:21,600 --> 00:04:24,410
for perception,
planning, and control.

105
00:04:24,410 --> 00:04:24,990
OK.

106
00:04:24,990 --> 00:04:30,322
So technically, I don't intend
to go into too many details,

107
00:04:30,322 --> 00:04:32,530
but I would love to answer
questions if you guys ask.

108
00:04:32,530 --> 00:04:35,520
And we can talk as much
as we want about it.

109
00:04:35,520 --> 00:04:38,950
But the overarching
theme to our approach

110
00:04:38,950 --> 00:04:41,250
when we're controlling,
perceiving, everything

111
00:04:41,250 --> 00:04:44,890
is to formulate everything
as an optimization problem.

112
00:04:44,890 --> 00:04:48,410
So even the simplest
example in robotics

113
00:04:48,410 --> 00:04:49,980
is the inverse
kinematics problem

114
00:04:49,980 --> 00:04:51,570
where you're just
trying to decide

115
00:04:51,570 --> 00:04:53,760
if I want to put my hand
in some particular place.

116
00:04:53,760 --> 00:04:56,555
I have to figure out if I have
a goal in the world coordinates.

117
00:04:56,555 --> 00:04:58,805
I have to figure out what
the joint coordinates should

118
00:04:58,805 --> 00:05:00,930
be to make that happen.

119
00:05:00,930 --> 00:05:03,660
So we have joint positions
in some vector q,

120
00:05:03,660 --> 00:05:06,270
and we just say, I'd like
to be as close as possible.

121
00:05:06,270 --> 00:05:09,827
I have some comfortable
position for my robot.

122
00:05:09,827 --> 00:05:11,910
We formulate the problem
as an optimization-- say,

123
00:05:11,910 --> 00:05:14,440
I'd like to be as close
to comfortable as possible

124
00:05:14,440 --> 00:05:17,250
in some simple cost function.

125
00:05:17,250 --> 00:05:19,440
And then I'm going to start
putting in constraints,

126
00:05:19,440 --> 00:05:23,100
like my hand is in the
desired configuration.

127
00:05:23,100 --> 00:05:25,190
But we have very
advanced constraints.

128
00:05:25,190 --> 00:05:26,899
So especially for the
balancing humanoid,

129
00:05:26,899 --> 00:05:28,606
we can say, for
instance, that the center

130
00:05:28,606 --> 00:05:30,430
mass has to be inside
the support polygon.

131
00:05:30,430 --> 00:05:32,430
We can say, we're about
to manipulate something.

132
00:05:32,430 --> 00:05:34,320
So I'd like the thing
I'm going to manipulate

133
00:05:34,320 --> 00:05:38,850
to be in the cone of visibility
of my vision sensors.

134
00:05:38,850 --> 00:05:40,220
I'd like my hand to approach.

135
00:05:40,220 --> 00:05:42,470
It doesn't matter where it
approaches along the table,

136
00:05:42,470 --> 00:05:45,570
maybe, but the palm should
be orthogonal to the table

137
00:05:45,570 --> 00:05:46,860
and should approach like this.

138
00:05:46,860 --> 00:05:50,521
And we put in more
and more sophisticated

139
00:05:50,521 --> 00:05:52,020
collision avoidance
type constraints

140
00:05:52,020 --> 00:05:54,810
and everything like this, and
the optimization framework

141
00:05:54,810 --> 00:05:59,631
as is general and can accept
those type of constraints.

142
00:05:59,631 --> 00:06:01,630
And then we can solve
them extremely efficiently

143
00:06:01,630 --> 00:06:04,592
with highly
optimized algorithms.

144
00:06:04,592 --> 00:06:06,300
So for instance, that
helped us with what

145
00:06:06,300 --> 00:06:08,500
I like to call the big
robot little car problem.

146
00:06:08,500 --> 00:06:10,100
So we have a very big robot.

147
00:06:10,100 --> 00:06:13,440
It's a 400 pound, six
foot something machine.

148
00:06:13,440 --> 00:06:15,810
And they asked us to
drive a very little car

149
00:06:15,810 --> 00:06:17,820
to the point where the
robot physically does not

150
00:06:17,820 --> 00:06:19,611
fit behind the steering
wheel-- impossible.

151
00:06:19,611 --> 00:06:21,540
It just doesn't kinematically.

152
00:06:21,540 --> 00:06:24,710
Torso's too big, steering
wheel's right there, no chance.

153
00:06:24,710 --> 00:06:26,760
So you have to drive
from the passenger seat.

154
00:06:26,760 --> 00:06:28,570
You have to put your
foot over the console.

155
00:06:28,570 --> 00:06:30,810
You have to drive like this,
and then our only option

156
00:06:30,810 --> 00:06:33,000
was to get out of
the passenger side.

157
00:06:33,000 --> 00:06:35,490
So that was a hard
problem kinematically,

158
00:06:35,490 --> 00:06:38,280
but we have this rich
library of optimizations.

159
00:06:38,280 --> 00:06:39,220
We can drag it around.

160
00:06:39,220 --> 00:06:41,910
We can explore different
kinematic configurations

161
00:06:41,910 --> 00:06:43,590
of the robot.

162
00:06:43,590 --> 00:06:46,080
But we also use the same
language of optimization

163
00:06:46,080 --> 00:06:49,260
and constraints, and then we
put in the dynamics of the robot

164
00:06:49,260 --> 00:06:50,580
as another constraint.

165
00:06:50,580 --> 00:06:53,430
And we can start doing
efficient dynamic motion

166
00:06:53,430 --> 00:06:55,060
planning with the same tools.

167
00:06:55,060 --> 00:06:57,360
So for instance, if we wanted
Atlas to suddenly start

168
00:06:57,360 --> 00:07:01,800
jumping off cinder blocks or
running, we did a lot of work

169
00:07:01,800 --> 00:07:03,990
in that regard to
make our optimization

170
00:07:03,990 --> 00:07:06,450
algorithms efficient
enough to scale

171
00:07:06,450 --> 00:07:08,100
to very complex
motions that could

172
00:07:08,100 --> 00:07:11,370
be planned on the fly
at interactive rates.

173
00:07:11,370 --> 00:07:16,200
So one of the things you
might be familiar with--

174
00:07:16,200 --> 00:07:19,170
Honda ASIMO is one of
the famous robots that

175
00:07:19,170 --> 00:07:23,610
walks around like this, and
it's a beautiful machine.

176
00:07:23,610 --> 00:07:27,240
They are extremely good at
real time planning using

177
00:07:27,240 --> 00:07:29,682
limiting assumptions
of keeping your center

178
00:07:29,682 --> 00:07:31,640
mass at a constant height
and things like this.

179
00:07:31,640 --> 00:07:33,619
And one of the
questions we asked is,

180
00:07:33,619 --> 00:07:35,160
can we take some of
the insights that

181
00:07:35,160 --> 00:07:37,080
have worked so well
on those robots

182
00:07:37,080 --> 00:07:40,530
and generalize them to
more general dynamic tasks?

183
00:07:40,530 --> 00:07:44,100
And one of the big ideas I want
to try to communicate quickly

184
00:07:44,100 --> 00:07:48,779
is that even though our robot
is extremely complicated,

185
00:07:48,779 --> 00:07:50,820
there's sort of a low
dimensional problem sitting

186
00:07:50,820 --> 00:07:52,590
inside the big high
dimensional problem.

187
00:07:52,590 --> 00:07:55,320
So if I start worrying about
every joint angle in my hand

188
00:07:55,320 --> 00:07:57,720
while I'm thinking about
walking, I'm dead, right?

189
00:07:57,720 --> 00:08:00,120
So actually, when you're
thinking about walking, even

190
00:08:00,120 --> 00:08:02,070
doing gymnastics or
something like this,

191
00:08:02,070 --> 00:08:03,870
I think the fundamental
representation

192
00:08:03,870 --> 00:08:07,020
is the dynamics of your
center of mass, your angular

193
00:08:07,020 --> 00:08:10,440
momentum, some bulk
dynamics of your robot,

194
00:08:10,440 --> 00:08:13,110
and the contact forces you're
exerting on the world, which

195
00:08:13,110 --> 00:08:14,539
are also constrained.

196
00:08:14,539 --> 00:08:16,080
And in this sort of
six dimensional--

197
00:08:16,080 --> 00:08:19,890
12 dimensional if you
have velocities-- space

198
00:08:19,890 --> 00:08:22,860
with these relatively
limited constraints,

199
00:08:22,860 --> 00:08:25,560
you can actually do
very efficient planning

200
00:08:25,560 --> 00:08:29,730
and then map that in a second
pass back to the full figure

201
00:08:29,730 --> 00:08:31,650
out what my pinky's going to do.

202
00:08:31,650 --> 00:08:35,204
So we do that.

203
00:08:35,204 --> 00:08:36,620
We spent a lot of
time doing that,

204
00:08:36,620 --> 00:08:40,559
and we can now plan motions
for complicated humanoids that

205
00:08:40,559 --> 00:08:43,539
were far beyond our ability
to do it a few years ago.

206
00:08:43,539 --> 00:08:45,410
This was a major effort for us.

207
00:08:45,410 --> 00:08:47,460
My kids and I were
watching American Ninja

208
00:08:47,460 --> 00:08:51,870
Warrior at the time, so we did
all the Ninja Warrior tasks.

209
00:08:51,870 --> 00:08:54,960
So there were some
algorithmic ideas

210
00:08:54,960 --> 00:08:56,580
that were required for that.

211
00:08:56,580 --> 00:08:58,950
It was also just a software
engineering exercise

212
00:08:58,950 --> 00:09:02,714
to build a dynamics engine that
provided analytical gradients,

213
00:09:02,714 --> 00:09:04,380
exposed all the
sparsity in the problem,

214
00:09:04,380 --> 00:09:06,330
and wrote custom solvers
and things like that

215
00:09:06,330 --> 00:09:08,081
to make that work.

216
00:09:08,081 --> 00:09:09,330
It's not just about humanoids.

217
00:09:09,330 --> 00:09:13,890
We spent a day after we got
Atlas doing those things

218
00:09:13,890 --> 00:09:16,470
to show that we could make
a quadruped run around using

219
00:09:16,470 --> 00:09:17,616
the same exact algorithms.

220
00:09:17,616 --> 00:09:18,990
It took literally
less than a day

221
00:09:18,990 --> 00:09:22,820
to make all these examples work.

222
00:09:22,820 --> 00:09:24,430
There's another
level of optimization

223
00:09:24,430 --> 00:09:26,260
that's kicking around in here.

224
00:09:26,260 --> 00:09:29,290
So the humanoid, in some
sense when it's moving around,

225
00:09:29,290 --> 00:09:32,390
is a fairly continuous
dynamical system.

226
00:09:32,390 --> 00:09:34,750
There's punctuations when
your foot hits the ground

227
00:09:34,750 --> 00:09:36,640
or something like
this, so you think

228
00:09:36,640 --> 00:09:39,770
of that as sort of a smooth
optimization problem.

229
00:09:39,770 --> 00:09:42,124
There's also a discrete
optimization problem

230
00:09:42,124 --> 00:09:43,790
sitting in there,
too, even for walking.

231
00:09:43,790 --> 00:09:46,614
So if you think
about it, the methods

232
00:09:46,614 --> 00:09:48,655
I just talked about--
we're really talking about,

233
00:09:48,655 --> 00:09:49,849
OK, I move like this.

234
00:09:49,849 --> 00:09:51,640
I would prefer to move
something like this,

235
00:09:51,640 --> 00:09:53,140
but there's a
continuum of solutions

236
00:09:53,140 --> 00:09:54,460
I could possibly take.

237
00:09:54,460 --> 00:09:57,499
For walking, there's also
this problem of just saying,

238
00:09:57,499 --> 00:10:00,040
am I going to move my right foot
first or my left foot first?

239
00:10:00,040 --> 00:10:03,540
Am I going to step on cinder
block one or cinder block two?

240
00:10:03,540 --> 00:10:05,122
There really is a
discrete problem

241
00:10:05,122 --> 00:10:06,580
which gives a
combinatorial problem

242
00:10:06,580 --> 00:10:09,070
if you have to make
long-term decisions on that.

243
00:10:09,070 --> 00:10:11,210
And one of the things
we've tried to do well

244
00:10:11,210 --> 00:10:14,680
is be very explicit about
modeling the discrete aspects

245
00:10:14,680 --> 00:10:16,690
and the continuous
aspects of the problem

246
00:10:16,690 --> 00:10:21,280
individually and using the
right solvers that could think

247
00:10:21,280 --> 00:10:23,420
about both of those together.

248
00:10:23,420 --> 00:10:27,670
So here's an example of how
we do interactive footstep

249
00:10:27,670 --> 00:10:28,850
planning with the robot.

250
00:10:28,850 --> 00:10:32,320
If it's standing in front of
some perceived cinder blocks,

251
00:10:32,320 --> 00:10:35,980
for instance, the human can
quickly label discrete regions

252
00:10:35,980 --> 00:10:38,200
just by moving a mouse around.

253
00:10:38,200 --> 00:10:42,352
The regions that come out are
actually fit by an algorithm.

254
00:10:42,352 --> 00:10:44,560
They look small, because
they're trying to figure out

255
00:10:44,560 --> 00:10:46,601
if the center of the foot
was inside that region,

256
00:10:46,601 --> 00:10:48,165
the whole foot
would fit on that.

257
00:10:48,165 --> 00:10:50,290
And they're also thinking
about balance constraints

258
00:10:50,290 --> 00:10:51,830
and other things like that.

259
00:10:51,830 --> 00:10:54,410
But now we have discrete
regions to possibly step in.

260
00:10:54,410 --> 00:10:56,770
We have a combinatorial
problem and the smooth problem

261
00:10:56,770 --> 00:10:58,640
of moving my center
of mass and the like,

262
00:10:58,640 --> 00:11:00,850
and we have very good
new solvers to do that.

263
00:11:00,850 --> 00:11:02,950
And seeded inside
that, I just want

264
00:11:02,950 --> 00:11:06,280
to communicate that there's all
these little technical nuggets.

265
00:11:06,280 --> 00:11:10,270
We had to find a new way to
make really fast approximations

266
00:11:10,270 --> 00:11:13,060
of big convex regions
of free space.

267
00:11:13,060 --> 00:11:15,700
So we have optimizations
that just figured out--

268
00:11:15,700 --> 00:11:17,230
the problem of
finding the biggest

269
00:11:17,230 --> 00:11:20,780
polygon that fits inside all
those obstacles is NP hard.

270
00:11:20,780 --> 00:11:22,520
We're not going to solve that.

271
00:11:22,520 --> 00:11:26,620
But it turns out finding
a pretty good polygon can

272
00:11:26,620 --> 00:11:28,450
be done extremely fast now.

273
00:11:28,450 --> 00:11:30,700
And the particular
way we did it scales

274
00:11:30,700 --> 00:11:33,610
to very high dimensions
and complicated obstacles

275
00:11:33,610 --> 00:11:36,430
to the point where we could
do it on raw sensor data,

276
00:11:36,430 --> 00:11:40,060
and that was an enabling
technology for us.

277
00:11:40,060 --> 00:11:43,090
So our robot now, when
it's making plans--

278
00:11:43,090 --> 00:11:46,060
so the one on the left is
just walking towards the goal.

279
00:11:46,060 --> 00:11:48,222
The one on the right, we
removed a cinder block.

280
00:11:48,222 --> 00:11:50,680
And normally, a robot would
kind of get confused and stuck,

281
00:11:50,680 --> 00:11:52,721
because it's just thinking
about this local plan,

282
00:11:52,721 --> 00:11:54,460
local plan, local plan.

283
00:11:54,460 --> 00:11:56,335
It wouldn't be able to
stop and go completely

284
00:11:56,335 --> 00:11:57,470
in the other direction.

285
00:11:57,470 --> 00:11:59,720
But now, since we have this
higher level combinatorial

286
00:11:59,720 --> 00:12:03,820
planning on top, we can make
these big, long-term decision

287
00:12:03,820 --> 00:12:07,930
making tasks at
interactive rates.

288
00:12:07,930 --> 00:12:10,390
Also, the robot was too
big to walk through a door,

289
00:12:10,390 --> 00:12:12,440
so we had to walk
sideways through a door.

290
00:12:12,440 --> 00:12:14,190
And that was sort of
a standing challenge.

291
00:12:14,190 --> 00:12:17,380
The guy who started the program
putting footsteps down by hand

292
00:12:17,380 --> 00:12:19,660
said, whatever I do
in footstep planning,

293
00:12:19,660 --> 00:12:22,780
I will never lay down footsteps
to walk through a door again.

294
00:12:22,780 --> 00:12:25,600
That was the challenge.

295
00:12:25,600 --> 00:12:27,520
We did a lot of work on
the balancing control

296
00:12:27,520 --> 00:12:30,670
for the robot, so it's
a force controlled robot

297
00:12:30,670 --> 00:12:33,670
using hydraulic
actuators everywhere.

298
00:12:33,670 --> 00:12:35,242
Again, I won't go
into the details,

299
00:12:35,242 --> 00:12:37,450
but we thought a lot about
the dynamics of the robot.

300
00:12:37,450 --> 00:12:40,720
How do you cast that as
an efficient optimization

301
00:12:40,720 --> 00:12:43,810
that we can solve on the fly?

302
00:12:43,810 --> 00:12:46,510
And we were solving an
optimization at a kilohertz

303
00:12:46,510 --> 00:12:48,370
to balance the robot.

304
00:12:48,370 --> 00:12:50,870
So you put it all together.

305
00:12:50,870 --> 00:12:53,650
And as a basic competency,
how well does our robot

306
00:12:53,650 --> 00:12:55,020
walk around and balance?

307
00:12:55,020 --> 00:12:59,260
Here's one of the
examples at a normal speed

308
00:12:59,260 --> 00:13:02,380
from the challenge.

309
00:13:02,380 --> 00:13:04,420
So the robot just puts
its footsteps down ahead.

310
00:13:04,420 --> 00:13:06,280
The operator is mostly
just watching and giving

311
00:13:06,280 --> 00:13:07,196
high level directions.

312
00:13:07,196 --> 00:13:09,820
I want to go over here, and the
robot's doing its own thing.

313
00:13:14,609 --> 00:13:16,150
Now, all the other
teams I know about

314
00:13:16,150 --> 00:13:20,440
were putting down the footsteps
by hand on the obstacles.

315
00:13:20,440 --> 00:13:23,530
I don't know if someone else
was doing it autonomously.

316
00:13:23,530 --> 00:13:26,530
We chose to do it autonomously.

317
00:13:26,530 --> 00:13:28,390
We were a little bit
faster because of it,

318
00:13:28,390 --> 00:13:34,110
but I don't know
if it was enabling.

319
00:13:34,110 --> 00:13:36,270
But very proud of our
walking, even though it's

320
00:13:36,270 --> 00:13:37,150
still conservative.

321
00:13:37,150 --> 00:13:39,530
This is lousy
compared to a human.

322
00:13:39,530 --> 00:13:41,674
Yeah?

323
00:13:41,674 --> 00:13:45,076
AUDIENCE: So the obstacles are
modeled by the robot's vision,

324
00:13:45,076 --> 00:13:46,899
or do you actually preset them?

325
00:13:46,899 --> 00:13:49,440
RUSS TEDRAKE: So we knew they
were going to be cinder blocks.

326
00:13:49,440 --> 00:13:51,840
We didn't know the orientation
or positions of them,

327
00:13:51,840 --> 00:13:53,946
so we had a cinder block
fitting algorithm that

328
00:13:53,946 --> 00:13:55,320
would run on the
fly, snap things

329
00:13:55,320 --> 00:13:58,050
into place with the cameras--

330
00:13:58,050 --> 00:13:59,970
actually, laser scanner.

331
00:13:59,970 --> 00:14:02,120
And then we walk up stairs.

332
00:14:02,120 --> 00:14:03,870
Little things-- if you
care about walking,

333
00:14:03,870 --> 00:14:06,750
the heels are
hanging off the back.

334
00:14:06,750 --> 00:14:08,310
There's special
algorithms in there

335
00:14:08,310 --> 00:14:11,010
to balance on partial foot
contact and things like that.

336
00:14:11,010 --> 00:14:12,910
And that made the difference.

337
00:14:12,910 --> 00:14:17,770
We could go up there
efficiently, robustly.

338
00:14:17,770 --> 00:14:20,980
So I would say though,
for conservative walking,

339
00:14:20,980 --> 00:14:22,750
it really works well.

340
00:14:22,750 --> 00:14:24,832
We could plant these
things on the fly.

341
00:14:24,832 --> 00:14:26,290
And we also had
this user interface

342
00:14:26,290 --> 00:14:28,624
that if the foot step planner
ever did something stupid,

343
00:14:28,624 --> 00:14:31,165
the human could just drag a foot
around, add a new constraint

344
00:14:31,165 --> 00:14:31,750
to the solver.

345
00:14:31,750 --> 00:14:34,150
It would continue to solve
with a new constraint

346
00:14:34,150 --> 00:14:37,000
and adjust its solutions.

347
00:14:37,000 --> 00:14:38,950
We could do more dynamic plans.

348
00:14:38,950 --> 00:14:40,910
We could have it run
everything like that.

349
00:14:40,910 --> 00:14:42,618
We actually never
tried this on the robot

350
00:14:42,618 --> 00:14:44,290
before the
competition, because we

351
00:14:44,290 --> 00:14:45,831
were terrified of
breaking the robot,

352
00:14:45,831 --> 00:14:47,561
and we couldn't
accept the downtime.

353
00:14:47,561 --> 00:14:49,060
But now that the
competition's over,

354
00:14:49,060 --> 00:14:52,240
this is exactly
what we're trying.

355
00:14:52,240 --> 00:14:55,650
But the optimizations are slower
and didn't always succeed.

356
00:14:55,650 --> 00:14:59,590
So in the real scenario, we were
putting some more constraints

357
00:14:59,590 --> 00:15:02,040
on and doing much more
conservative gaits.

358
00:15:02,040 --> 00:15:04,540
The balance control I'd
say worked extremely well.

359
00:15:04,540 --> 00:15:08,680
So the hardest task was this
getting out of the car task.

360
00:15:08,680 --> 00:15:09,790
We worked like crazy.

361
00:15:09,790 --> 00:15:11,380
We didn't work on
it until the end.

362
00:15:11,380 --> 00:15:13,540
I thought DARPA was going
to scratch it, honestly.

363
00:15:13,540 --> 00:15:16,674
But in the last month, it became
clear that we had to do it.

364
00:15:16,674 --> 00:15:18,340
And then we spent a
lot of effort on it.

365
00:15:18,340 --> 00:15:21,097
And we put the car in
every possible situation.

366
00:15:21,097 --> 00:15:22,180
This was on cinder blocks.

367
00:15:22,180 --> 00:15:23,800
It's way high.

368
00:15:23,800 --> 00:15:26,580
It has to step down almost
beyond its reachability

369
00:15:26,580 --> 00:15:27,790
in the leg.

370
00:15:27,790 --> 00:15:30,340
This thing was just super solid.

371
00:15:30,340 --> 00:15:32,410
So Andres and Lucas
were the main designers

372
00:15:32,410 --> 00:15:33,910
of this algorithm.

373
00:15:33,910 --> 00:15:36,970
I'd say it's superhuman
in this regard, right?

374
00:15:36,970 --> 00:15:39,230
A human would not
do that, of course,

375
00:15:39,230 --> 00:15:41,530
but standing on one
foot while someone's

376
00:15:41,530 --> 00:15:45,075
jumping on the car like this--

377
00:15:45,075 --> 00:15:46,305
it really works well.

378
00:15:46,305 --> 00:15:48,430
In fact, the hardest part
of that for the algorithm

379
00:15:48,430 --> 00:15:51,346
was the fact that it's trying
to find out where the ground is,

380
00:15:51,346 --> 00:15:52,720
and the camera's
going like this.

381
00:15:52,720 --> 00:15:55,420
So that was the reason
it had this long pause

382
00:15:55,420 --> 00:15:58,120
before it went down.

383
00:15:58,120 --> 00:16:04,012
But there was one time
that it didn't work well,

384
00:16:04,012 --> 00:16:05,470
and it's hard for
me to watch this.

385
00:16:05,470 --> 00:16:11,590
But it turns out on the first--
you saw that little kick?

386
00:16:11,590 --> 00:16:13,060
This was horrible.

387
00:16:15,818 --> 00:16:17,917
I'll tell you exactly
what happened,

388
00:16:17,917 --> 00:16:19,750
but I think it really
exposed the limitation

389
00:16:19,750 --> 00:16:22,660
of the state of the art.

390
00:16:22,660 --> 00:16:26,710
So what happened in that
particular situation was

391
00:16:26,710 --> 00:16:30,310
the robot was almost
autonomous in some ways,

392
00:16:30,310 --> 00:16:33,010
and we basically tried
to have the human

393
00:16:33,010 --> 00:16:34,420
have to do almost nothing.

394
00:16:34,420 --> 00:16:36,880
And in the end, we got
the humans checklist

395
00:16:36,880 --> 00:16:40,000
down to about five items,
which was probably a mistake,

396
00:16:40,000 --> 00:16:42,100
because we screwed
up on the checklist.

397
00:16:42,100 --> 00:16:45,120
So one of the five
items was to--

398
00:16:45,120 --> 00:16:47,260
we have one set of programs
that are running when

399
00:16:47,260 --> 00:16:49,150
the robot's driving the car.

400
00:16:49,150 --> 00:16:51,335
And then all the
human had to do was

401
00:16:51,335 --> 00:16:53,710
turn off the driving controller
and turn on the balancing

402
00:16:53,710 --> 00:16:54,670
controller.

403
00:16:54,670 --> 00:16:57,130
But it was exciting and the
first day of the competition.

404
00:16:57,130 --> 00:16:59,752
And we turned on the balancing
controller, forgot to turn off

405
00:16:59,752 --> 00:17:00,710
the driving controller.

406
00:17:00,710 --> 00:17:04,450
So the ankle was still
trying to drive the car.

407
00:17:04,450 --> 00:17:06,290
Even that, the controller
was robust enough.

408
00:17:06,290 --> 00:17:10,300
So I really think there's
this fundamental thing

409
00:17:10,300 --> 00:17:12,430
that if you're close
to your nominal plan,

410
00:17:12,430 --> 00:17:13,750
things were very robust.

411
00:17:13,750 --> 00:17:16,136
But what happened is the ankle
is still driving the car.

412
00:17:16,136 --> 00:17:18,010
I think we could balance
with the ankle doing

413
00:17:18,010 --> 00:17:20,650
the wrong thing, except the
ankle did the wrong thing

414
00:17:20,650 --> 00:17:24,550
just enough that the tailbone
hit the seat of the car.

415
00:17:24,550 --> 00:17:27,550
That was no longer something
we could handle, right?

416
00:17:27,550 --> 00:17:30,860
So there was no contact
sensor in the butt.

417
00:17:30,860 --> 00:17:33,300
That meant the dynamics
model was very wrong.

418
00:17:33,300 --> 00:17:35,090
The state estimator
got very confused.

419
00:17:35,090 --> 00:17:36,790
The foot came off the ground,
and the state estimator

420
00:17:36,790 --> 00:17:39,123
had an assumption that the
feet should be on the ground.

421
00:17:39,123 --> 00:17:41,500
That's how it knew where
it was in the world.

422
00:17:41,500 --> 00:17:43,930
And basically, the
controller was hosed, right?

423
00:17:43,930 --> 00:17:46,971
And that was the only time we
could have done that badly--

424
00:17:46,971 --> 00:17:48,220
the vibrations and everything.

425
00:17:48,220 --> 00:17:50,482
I had emails from people
of all walks of life

426
00:17:50,482 --> 00:17:52,690
telling me what they thought
was wrong with the brain

427
00:17:52,690 --> 00:17:56,140
of the robot from
shaking like that.

428
00:17:56,140 --> 00:17:57,560
But that was a bad thing.

429
00:17:57,560 --> 00:17:59,970
So you know I think
fundamentally,

430
00:17:59,970 --> 00:18:01,726
if we're thinking about plans--

431
00:18:01,726 --> 00:18:03,100
and that's what
we know how to do

432
00:18:03,100 --> 00:18:07,772
at scale for high dimensional
systems is single solutions--

433
00:18:07,772 --> 00:18:08,980
then we're close to the plan.

434
00:18:08,980 --> 00:18:09,646
Things are good.

435
00:18:09,646 --> 00:18:12,160
When we're far from the
plan, we're not very good.

436
00:18:12,160 --> 00:18:15,370
And a change in the
contact situation--

437
00:18:15,370 --> 00:18:18,710
even if it's in a Cartesian
space very close--

438
00:18:18,710 --> 00:18:22,180
change in the contact situation
is a big change to the plan.

439
00:18:22,180 --> 00:18:24,580
There's lots of
ways to address it.

440
00:18:24,580 --> 00:18:26,784
We're doing all of them now.

441
00:18:26,784 --> 00:18:28,450
It's all fundamentally
about robustness.

442
00:18:28,450 --> 00:18:31,069
But ironically, the
car was the only time

443
00:18:31,069 --> 00:18:32,610
we could have done
that badly, right?

444
00:18:32,610 --> 00:18:35,770
So every other place, we
worked out all these situations

445
00:18:35,770 --> 00:18:38,830
where, OK, the robot's walking,
and then something bad happens

446
00:18:38,830 --> 00:18:42,220
and someone lances
you or something.

447
00:18:42,220 --> 00:18:43,420
We had recovery.

448
00:18:43,420 --> 00:18:45,640
And then even if it
tried to take a step--

449
00:18:45,640 --> 00:18:47,920
even if that failed, it
would go into a gentle mode

450
00:18:47,920 --> 00:18:49,090
where it would
protect its hands,

451
00:18:49,090 --> 00:18:50,130
because we were afraid
of breaking the hands.

452
00:18:50,130 --> 00:18:52,300
It would fall very
generally to the ground.

453
00:18:52,300 --> 00:18:54,070
All that was good.

454
00:18:54,070 --> 00:18:56,320
We turned it off exactly
once in the competition.

455
00:18:56,320 --> 00:18:57,490
We turned it off when
we were in the car,

456
00:18:57,490 --> 00:18:59,800
because we can't take a step to
recover when you're in the car

457
00:18:59,800 --> 00:19:01,420
and you're the same
size of the car.

458
00:19:01,420 --> 00:19:03,430
And we didn't even want
to protect our hands,

459
00:19:03,430 --> 00:19:07,532
because once we got our hand
stuck on the steering wheel.

460
00:19:07,532 --> 00:19:08,990
So anyways, that
was the only thing

461
00:19:08,990 --> 00:19:12,041
we could have sort of shaken
ourselves silly and fallen.

462
00:19:12,041 --> 00:19:12,790
And what happened?

463
00:19:12,790 --> 00:19:15,220
We fell down with
our 400 pound robot.

464
00:19:15,220 --> 00:19:17,630
We broke the arm--

465
00:19:17,630 --> 00:19:18,970
the right arm.

466
00:19:18,970 --> 00:19:21,360
Sadly, all of our
practices ever were

467
00:19:21,360 --> 00:19:23,821
doing all the
tasks right handed,

468
00:19:23,821 --> 00:19:26,070
but we got to show off a
different form of robustness.

469
00:19:26,070 --> 00:19:29,250
So actually, because we had so
much autonomy in the system,

470
00:19:29,250 --> 00:19:31,920
we flipped a bit
and said, let's use

471
00:19:31,920 --> 00:19:33,150
the left arm for everything.

472
00:19:33,150 --> 00:19:36,506
Which is more than just map the
joint coordinates over here.

473
00:19:36,506 --> 00:19:38,130
It meant you had to
walk up to the door

474
00:19:38,130 --> 00:19:40,470
on the other side of the door.

475
00:19:40,470 --> 00:19:43,680
The implications
back up quite a bit.

476
00:19:43,680 --> 00:19:46,970
After having our arm
just completely hosed,

477
00:19:46,970 --> 00:19:50,310
we were able to go through and
do all the rest of the tasks

478
00:19:50,310 --> 00:19:52,460
except for the drill,
which required two hands.

479
00:19:52,460 --> 00:19:53,460
We couldn't do that one.

480
00:19:53,460 --> 00:19:55,290
We had to pick up the
drill and turn it on.

481
00:19:55,290 --> 00:19:58,650
So we ended the
day in second place

482
00:19:58,650 --> 00:20:01,176
with a different
display of robustness.

483
00:20:01,176 --> 00:20:02,856
AUDIENCE: That's still
pretty damn good.

484
00:20:02,856 --> 00:20:04,730
RUSS TEDRAKE: We were
happy, but not as happy

485
00:20:04,730 --> 00:20:05,729
as if we had not fallen.

486
00:20:08,050 --> 00:20:08,550
OK.

487
00:20:08,550 --> 00:20:11,829
So I think walking around,
balancing-- we're pretty good,

488
00:20:11,829 --> 00:20:12,870
but there's a limitation.

489
00:20:12,870 --> 00:20:18,070
I really do think everybody has
that limitation to some extent.

490
00:20:18,070 --> 00:20:19,820
The manipulation
capabilities of the robot

491
00:20:19,820 --> 00:20:23,060
were pretty limited, just
because we didn't need

492
00:20:23,060 --> 00:20:24,920
to do it for the challenge.

493
00:20:24,920 --> 00:20:27,670
The manipulation
requirements were minimal.

494
00:20:27,670 --> 00:20:29,060
You had to open doors.

495
00:20:29,060 --> 00:20:32,150
Picking up a drill was the
most complicated thing.

496
00:20:32,150 --> 00:20:34,820
We actually had a lot of
really nice robotic hands

497
00:20:34,820 --> 00:20:38,580
to play with, but they all broke
when you started really running

498
00:20:38,580 --> 00:20:39,920
them through these hard tests.

499
00:20:39,920 --> 00:20:42,950
So we ended up with these
sort of lobster claw

500
00:20:42,950 --> 00:20:44,940
kind of grippers, because
they didn't break.

501
00:20:44,940 --> 00:20:46,790
And they were robust,
and they worked well.

502
00:20:46,790 --> 00:20:50,420
But it limited what we
could do in manipulation.

503
00:20:50,420 --> 00:20:53,030
Again, the planning
worked very well.

504
00:20:53,030 --> 00:20:54,950
We could pick up
a board and even

505
00:20:54,950 --> 00:20:58,490
plan to make sure that the
board now didn't intersect

506
00:20:58,490 --> 00:20:59,810
with other boards in the world.

507
00:20:59,810 --> 00:21:02,120
And we have really good
planning capabilities,

508
00:21:02,120 --> 00:21:06,230
and those worked at interactive
rates-- the kinematic plans.

509
00:21:06,230 --> 00:21:09,330
But the grasping was open loop,
so there's really no feedback.

510
00:21:09,330 --> 00:21:12,800
So there's current sensing
just to not overheat the hands.

511
00:21:12,800 --> 00:21:14,780
But basically, you
do a lot of thinking

512
00:21:14,780 --> 00:21:17,120
to figure out how to get
your hand near the board.

513
00:21:17,120 --> 00:21:19,369
And then you kind of close
your eyes and go like this,

514
00:21:19,369 --> 00:21:20,810
and hope it lands in the hand.

515
00:21:20,810 --> 00:21:22,070
And most of the time, it does.

516
00:21:22,070 --> 00:21:24,890
Every once in
awhile, it doesn't.

517
00:21:24,890 --> 00:21:26,990
We experimented with
every touch sensor

518
00:21:26,990 --> 00:21:28,230
we could get our hands on.

519
00:21:28,230 --> 00:21:29,930
That wasn't meant to be a pun.

520
00:21:29,930 --> 00:21:31,460
And we tried cameras
and everything,

521
00:21:31,460 --> 00:21:33,800
but they were all just too
fragile and difficult to use

522
00:21:33,800 --> 00:21:36,091
for the competition.

523
00:21:36,091 --> 00:21:38,090
We're doing a lot of work
now doing optimization

524
00:21:38,090 --> 00:21:42,260
for grasping, but I'll
skip over that for time.

525
00:21:42,260 --> 00:21:44,060
So the other piece
was, how does the human

526
00:21:44,060 --> 00:21:47,360
come into the perception
side of the story?

527
00:21:47,360 --> 00:21:50,690
So one of these tasks
was moving debris

528
00:21:50,690 --> 00:21:52,430
out from in front of a door.

529
00:21:52,430 --> 00:21:56,360
This is what it looked like
in the original version

530
00:21:56,360 --> 00:21:57,700
of the competition-- the trials.

531
00:21:57,700 --> 00:21:59,150
The robot would come up
and throw these boards out

532
00:21:59,150 --> 00:22:01,580
of the way, and you see the
human operators over there

533
00:22:01,580 --> 00:22:05,850
with her big
console of displays.

534
00:22:05,850 --> 00:22:08,360
This is what the laser
in the robot's head sees.

535
00:22:08,360 --> 00:22:09,950
We have a spinning laser.

536
00:22:09,950 --> 00:22:11,790
We also have stereo vision.

537
00:22:11,790 --> 00:22:14,000
But the laser
reconstruction of this

538
00:22:14,000 --> 00:22:15,621
gives you a mess of points.

539
00:22:15,621 --> 00:22:17,495
If you asked a vision
algorithm-- some of you

540
00:22:17,495 --> 00:22:19,280
are vision experts
I'm sure in the room.

541
00:22:19,280 --> 00:22:20,810
If you asked a vision
algorithm to figure out

542
00:22:20,810 --> 00:22:22,490
what's going on in
that mess of points,

543
00:22:22,490 --> 00:22:24,917
it's an extremely hard problem.

544
00:22:24,917 --> 00:22:26,250
But we have a human in the loop.

545
00:22:26,250 --> 00:22:29,870
So the idea is that one
or two clicks from a human

546
00:22:29,870 --> 00:22:31,700
can turn that from an
intractable problem

547
00:22:31,700 --> 00:22:33,350
to a pretty simple problem.

548
00:22:33,350 --> 00:22:35,420
Just say, there's
a two by four here.

549
00:22:35,420 --> 00:22:38,870
And then now a local search
can do sort of RANSAC type

550
00:22:38,870 --> 00:22:42,050
local optimizations to
find the best fit to a two

551
00:22:42,050 --> 00:22:45,712
by four to that local group of
points, and that works well.

552
00:22:45,712 --> 00:22:48,170
And so the robot didn't have
to think about the messy point

553
00:22:48,170 --> 00:22:49,742
clouds when it's
doing its planning.

554
00:22:49,742 --> 00:22:51,200
It could think
about the simplified

555
00:22:51,200 --> 00:22:53,480
geometry from the CAD models.

556
00:22:53,480 --> 00:22:56,689
And most of the planning
was just on the CAD models.

557
00:22:56,689 --> 00:22:58,730
So this is what it looks
like to drive the robot.

558
00:22:58,730 --> 00:23:01,900
So you click somewhere
saying there's a valve,

559
00:23:01,900 --> 00:23:03,774
then the perception
algorithm finds a valve.

560
00:23:03,774 --> 00:23:04,940
Then the robot starts going.

561
00:23:04,940 --> 00:23:08,900
It actually shows you a ghost
of what it's about to do.

562
00:23:08,900 --> 00:23:10,570
And then if you're
happy with it,

563
00:23:10,570 --> 00:23:12,939
and if all things are
going well, you just watch.

564
00:23:12,939 --> 00:23:15,230
But if it looks like it's
about to do something stupid,

565
00:23:15,230 --> 00:23:18,140
you can come in, stop,
interact, change the plans,

566
00:23:18,140 --> 00:23:21,380
and let it do its thing.

567
00:23:21,380 --> 00:23:24,950
It's kind of fun to watch the
robot view of the world, right?

568
00:23:24,950 --> 00:23:27,402
So this is what the robot sees.

569
00:23:27,402 --> 00:23:28,610
It throws down its footsteps.

570
00:23:28,610 --> 00:23:30,980
It's deciding how to
walk up to that valve.

571
00:23:30,980 --> 00:23:34,280
Again, when the
right arm was broken,

572
00:23:34,280 --> 00:23:35,870
this was one of
our practice runs.

573
00:23:35,870 --> 00:23:38,000
The right arm was
broken, it had a valve.

574
00:23:38,000 --> 00:23:39,230
We had to bit flip, and
now it had to walk over

575
00:23:39,230 --> 00:23:40,521
to the other side of the valve.

576
00:23:40,521 --> 00:23:42,110
And there's a lot
of things going on.

577
00:23:42,110 --> 00:23:44,776
A lot of pieces had to work well
together to make all this work.

578
00:23:47,350 --> 00:23:51,220
One of the questions that
I'll get before you ask it.

579
00:23:51,220 --> 00:23:53,050
If you've written it
down, OK, that's fine.

580
00:23:53,050 --> 00:23:54,280
Why were the robots so slow?

581
00:23:54,280 --> 00:23:56,350
Why were they standing still?

582
00:23:56,350 --> 00:24:00,190
A lot of people out there
waiting for the human, maybe,

583
00:24:00,190 --> 00:24:01,075
but for us it wasn't.

584
00:24:01,075 --> 00:24:02,560
It wasn't the planning time.

585
00:24:02,560 --> 00:24:04,690
The planning algorithms
were super fast.

586
00:24:04,690 --> 00:24:07,090
Most of the time, we were
waiting for sensor data.

587
00:24:07,090 --> 00:24:08,590
And that meant there
was two things.

588
00:24:08,590 --> 00:24:11,500
There was waiting for the
laser to spin completely around

589
00:24:11,500 --> 00:24:13,600
and also just being
conservative-- wanting

590
00:24:13,600 --> 00:24:16,120
to get that laser data
while the robot was stopped.

591
00:24:16,120 --> 00:24:18,400
And then there was getting
the laser data back

592
00:24:18,400 --> 00:24:20,230
to the computer that
had the fast planning

593
00:24:20,230 --> 00:24:21,320
algorithms in back.

594
00:24:21,320 --> 00:24:22,240
So if there was a
network blackout,

595
00:24:22,240 --> 00:24:24,010
we had to wait a
little bit, and that

596
00:24:24,010 --> 00:24:25,840
meant we were standing still.

597
00:24:25,840 --> 00:24:28,180
But we've actually done
a lot of work in lab

598
00:24:28,180 --> 00:24:30,730
to show that we don't
have to stand still.

599
00:24:30,730 --> 00:24:33,460
This is now the robot
walking with its laser

600
00:24:33,460 --> 00:24:36,820
blindfolded and using
only stereo vision

601
00:24:36,820 --> 00:24:40,000
using one of the capabilities
that came out of John's lab

602
00:24:40,000 --> 00:24:43,990
and others to do stereo fusion.

603
00:24:43,990 --> 00:24:47,290
The laser gives very
accurate points,

604
00:24:47,290 --> 00:24:49,759
but it gives them
slowly at a low rate.

605
00:24:49,759 --> 00:24:51,550
And you have to wait
for it to spin around.

606
00:24:51,550 --> 00:24:55,979
The camera is very dense, very
high rate, but very noisy.

607
00:24:55,979 --> 00:24:58,270
And John and others have
developed these new algorithms

608
00:24:58,270 --> 00:25:00,970
that can do real time
filtering of that noisy data,

609
00:25:00,970 --> 00:25:03,430
and we demonstrated that
they were good enough

610
00:25:03,430 --> 00:25:04,539
to do walking on.

611
00:25:04,539 --> 00:25:06,580
And so we put all the
pieces together-- real time

612
00:25:06,580 --> 00:25:08,680
footstep planning, real
time balancing, real time

613
00:25:08,680 --> 00:25:11,305
perception-- and we were able to
show we can walk continuously.

614
00:25:11,305 --> 00:25:12,880
This will be the future.

615
00:25:12,880 --> 00:25:14,972
So we had to do networking.

616
00:25:14,972 --> 00:25:16,180
We optimized network systems.

617
00:25:16,180 --> 00:25:19,360
We had to build servers, unit
test logistics, politics.

618
00:25:19,360 --> 00:25:22,270
It was exhausting.

619
00:25:22,270 --> 00:25:24,880
I think it was overall
incredibly good

620
00:25:24,880 --> 00:25:26,780
experience-- a huge success.

621
00:25:26,780 --> 00:25:28,690
I think the robots
can move faster

622
00:25:28,690 --> 00:25:31,915
with only small changes
mostly on the perception side.

623
00:25:31,915 --> 00:25:33,040
The walking was sufficient.

624
00:25:33,040 --> 00:25:35,677
We can definitely do better.

625
00:25:35,677 --> 00:25:37,010
The manipulation was very basic.

626
00:25:37,010 --> 00:25:38,468
I think we need to
do better there,

627
00:25:38,468 --> 00:25:40,750
but we didn't have
to for those tasks.

628
00:25:40,750 --> 00:25:44,500
The robustness
dominated everything.

629
00:25:44,500 --> 00:25:46,660
So I'll just end
and take questions,

630
00:25:46,660 --> 00:25:49,660
but I'll show this sort
of fun, again, robot view.

631
00:25:49,660 --> 00:25:53,430
This is the robot's God's
eye view of the world

632
00:25:53,430 --> 00:25:54,910
while it's doing
all these tasks.

633
00:25:54,910 --> 00:25:58,810
You can sort of see what the
robot labels with the geometry

634
00:25:58,810 --> 00:26:01,199
and what it's
leading its points.

635
00:26:01,199 --> 00:26:03,490
And it's just kind of fun to
have on in the background,

636
00:26:03,490 --> 00:26:06,190
and then I'll take
any questions.

637
00:26:06,190 --> 00:26:09,240
[APPLAUSE]