WEBVTT 00:00:00.000 --> 00:00:02.490 align:middle line:84% The following content is provided under a Creative 00:00:02.490 --> 00:00:03.630 align:middle line:90% Commons license. 00:00:03.630 --> 00:00:06.120 align:middle line:84% Your support will help MIT OpenCourseWare 00:00:06.120 --> 00:00:09.990 align:middle line:84% continue to offer high quality educational resources for free. 00:00:09.990 --> 00:00:12.720 align:middle line:84% To make a donation or to view additional materials 00:00:12.720 --> 00:00:16.650 align:middle line:84% from hundreds of MIT courses, visit MIT OpenCourseWare 00:00:16.650 --> 00:00:20.118 align:middle line:90% at ocw.mit.edu. 00:00:20.118 --> 00:00:21.392 align:middle line:90% [SIDE CONVERSATIONS] 00:00:21.392 --> 00:00:22.350 align:middle line:90% MICAH KIMO JOHNSON: OK. 00:00:22.350 --> 00:00:23.580 align:middle line:90% Let's get started, again. 00:00:23.580 --> 00:00:26.730 align:middle line:90% [SIDE CONVERSATIONS] 00:00:26.730 --> 00:00:30.870 align:middle line:84% I'm going to tell you guys about a recent project that I did, 00:00:30.870 --> 00:00:34.800 align:middle line:84% which was presented at CVPR this summer. 00:00:34.800 --> 00:00:39.370 align:middle line:84% It was a paper there and a demo, and I also 00:00:39.370 --> 00:00:41.340 align:middle line:84% was part of Siggraph Emerging Technologies, 00:00:41.340 --> 00:00:44.430 align:middle line:84% and it was pretty well received there too. 00:00:44.430 --> 00:00:47.352 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:00:47.352 --> 00:00:48.810 align:middle line:84% MICAH KIMO JOHNSON: We're starting. 00:00:48.810 --> 00:00:50.855 align:middle line:90% AUDIENCE: OK. 00:00:50.855 --> 00:00:52.230 align:middle line:84% MICAH KIMO JOHNSON: The main idea 00:00:52.230 --> 00:00:56.310 align:middle line:84% is a technique for capturing geometry. 00:00:56.310 --> 00:00:58.230 align:middle line:84% I have here, this is our little pocket 00:00:58.230 --> 00:01:03.360 align:middle line:84% demo, which I'll pass around, but it's a really simple idea. 00:01:03.360 --> 00:01:05.519 align:middle line:84% This is a piece of gel with a painted skin, 00:01:05.519 --> 00:01:09.120 align:middle line:84% and if you press something into it, like your finger, 00:01:09.120 --> 00:01:11.130 align:middle line:90% it looks like it's been painted. 00:01:11.130 --> 00:01:12.810 align:middle line:90% So I'll pass this around. 00:01:12.810 --> 00:01:15.030 align:middle line:84% You just take turns pressing your finger into it, 00:01:15.030 --> 00:01:16.830 align:middle line:90% looking at it. 00:01:16.830 --> 00:01:18.780 align:middle line:84% It's pretty simple and pretty cool, 00:01:18.780 --> 00:01:20.640 align:middle line:84% and what I'll tell you about in this talk 00:01:20.640 --> 00:01:24.300 align:middle line:84% is how we can actually take advantage of this simple device 00:01:24.300 --> 00:01:28.140 align:middle line:90% to measure surface geometry. 00:01:28.140 --> 00:01:32.160 align:middle line:84% Now this is been a pretty long standing problem in vision, 00:01:32.160 --> 00:01:34.620 align:middle line:90% how can you capture geometry? 00:01:34.620 --> 00:01:36.780 align:middle line:84% How can you get more than a 2D image? 00:01:36.780 --> 00:01:39.480 align:middle line:84% For example, how do you get 3D shape? 00:01:39.480 --> 00:01:42.780 align:middle line:84% And I have some references here, probably not so 00:01:42.780 --> 00:01:45.030 align:middle line:90% interesting for this class. 00:01:45.030 --> 00:01:48.540 align:middle line:84% Basically, for these different techniques, 00:01:48.540 --> 00:01:50.130 align:middle line:84% they are thinking more generally. 00:01:50.130 --> 00:01:54.660 align:middle line:84% How do I capture geometry of somewhat general objects? 00:01:54.660 --> 00:01:55.410 align:middle line:90% And I'll show you. 00:01:55.410 --> 00:01:57.390 align:middle line:84% This has some very specific properties. 00:01:57.390 --> 00:02:00.840 align:middle line:84% It's only going to be able to capture surface texture 00:02:00.840 --> 00:02:04.230 align:middle line:84% and surface geometry, but we still think it's pretty cool. 00:02:04.230 --> 00:02:07.470 align:middle line:90% 00:02:07.470 --> 00:02:10.014 align:middle line:84% For example, it can capture geometries 00:02:10.014 --> 00:02:10.889 align:middle line:90% of objects like this. 00:02:10.889 --> 00:02:13.260 align:middle line:84% We have an Oreo cookie here, so the surface of it 00:02:13.260 --> 00:02:15.930 align:middle line:90% has some interesting detail. 00:02:15.930 --> 00:02:19.110 align:middle line:90% This is a glass little trinket. 00:02:19.110 --> 00:02:20.520 align:middle line:90% It's like a pin. 00:02:20.520 --> 00:02:26.190 align:middle line:84% And a lot of existing methods that either use lasers or use 00:02:26.190 --> 00:02:28.830 align:middle line:84% structured light have problems with glass, 00:02:28.830 --> 00:02:32.220 align:middle line:84% so that's a difficult type of object to scan. 00:02:32.220 --> 00:02:34.710 align:middle line:90% This is a metal drawer pole. 00:02:34.710 --> 00:02:36.180 align:middle line:84% It's just a small piece of metal, 00:02:36.180 --> 00:02:40.320 align:middle line:84% but techniques that use cameras and structure light 00:02:40.320 --> 00:02:41.910 align:middle line:84% also have problems with metal because 00:02:41.910 --> 00:02:43.980 align:middle line:90% of the reflectance function. 00:02:43.980 --> 00:02:49.620 align:middle line:84% And we can even scan the surface geometry of something 00:02:49.620 --> 00:02:51.780 align:middle line:84% as fine as a $20 bill, so I'll show 00:02:51.780 --> 00:02:54.303 align:middle line:90% a lot of different results. 00:02:54.303 --> 00:02:55.470 align:middle line:90% Here's the main idea, again. 00:02:55.470 --> 00:02:59.100 align:middle line:84% You have some of this clear gel, called an elastomer, 00:02:59.100 --> 00:03:00.540 align:middle line:90% thermoplastic elastomer. 00:03:00.540 --> 00:03:05.490 align:middle line:84% We put some paint on top, when you press something in and view 00:03:05.490 --> 00:03:08.430 align:middle line:84% it from the other side, it looks like it's been painted. 00:03:08.430 --> 00:03:10.110 align:middle line:84% And when you get the pocket version, 00:03:10.110 --> 00:03:12.690 align:middle line:90% you can see that yourself. 00:03:12.690 --> 00:03:16.710 align:middle line:84% So we can use standard computer vision techniques 00:03:16.710 --> 00:03:19.920 align:middle line:84% to reconstruct the surface geometry, 00:03:19.920 --> 00:03:23.910 align:middle line:84% and I'll tell you about that in the rest of the talk. 00:03:23.910 --> 00:03:27.120 align:middle line:84% But first, what is this material? 00:03:27.120 --> 00:03:30.480 align:middle line:84% Well, it can be any kind of clear gel, 00:03:30.480 --> 00:03:34.350 align:middle line:84% so we use thermoplastic elastomer mostly, 00:03:34.350 --> 00:03:36.750 align:middle line:84% which is a common rubber that people haven't heard of, 00:03:36.750 --> 00:03:39.390 align:middle line:84% but it's the same kind of rubber that's in your shoes 00:03:39.390 --> 00:03:42.870 align:middle line:84% or in a lot of different products you buy. 00:03:42.870 --> 00:03:45.420 align:middle line:84% But you can also use silicones and other rubbers. 00:03:45.420 --> 00:03:48.720 align:middle line:84% It's just important that it's clear. 00:03:48.720 --> 00:03:51.150 align:middle line:90% And we bake it in the oven. 00:03:51.150 --> 00:03:53.910 align:middle line:84% We melt it down into whatever shape we want, 00:03:53.910 --> 00:03:58.138 align:middle line:84% and typically, we want it to be flat on top. 00:03:58.138 --> 00:04:00.180 align:middle line:84% But you could make it into the shape of your face 00:04:00.180 --> 00:04:02.160 align:middle line:84% because you can use this stuff to make molds, 00:04:02.160 --> 00:04:04.388 align:middle line:84% so just some interesting things we've done making it 00:04:04.388 --> 00:04:05.430 align:middle line:90% to other types of shapes. 00:04:05.430 --> 00:04:09.840 align:middle line:84% But typically for this purpose, we want it flat. 00:04:09.840 --> 00:04:12.120 align:middle line:84% We engineer a specific paint to put on top, 00:04:12.120 --> 00:04:15.150 align:middle line:84% and you can see these are four different sensors, 00:04:15.150 --> 00:04:17.010 align:middle line:90% four different paints. 00:04:17.010 --> 00:04:19.800 align:middle line:84% And the properties of the paint kind of 00:04:19.800 --> 00:04:22.680 align:middle line:84% change the types of geometries you can measure. 00:04:22.680 --> 00:04:25.398 align:middle line:84% I'll maybe talk about that a little more. 00:04:25.398 --> 00:04:27.690 align:middle line:84% But what we do is we put this into a photometric stereo 00:04:27.690 --> 00:04:30.270 align:middle line:90% setup. 00:04:30.270 --> 00:04:31.830 align:middle line:90% There's some gel on the top. 00:04:31.830 --> 00:04:34.020 align:middle line:90% This is a box that I made. 00:04:34.020 --> 00:04:36.870 align:middle line:84% And you've got a camera beneath it. 00:04:36.870 --> 00:04:38.670 align:middle line:84% And then inside, there's a Tupperware bowl 00:04:38.670 --> 00:04:39.900 align:middle line:84% with three lights, red, green, and blue 00:04:39.900 --> 00:04:41.220 align:middle line:90% from different directions. 00:04:41.220 --> 00:04:43.950 align:middle line:90% 00:04:43.950 --> 00:04:45.975 align:middle line:84% Let me show you actually a video. 00:04:45.975 --> 00:04:48.530 align:middle line:90% 00:04:48.530 --> 00:04:51.920 align:middle line:84% So as part of the Siggraph presentation, 00:04:51.920 --> 00:04:55.190 align:middle line:84% I actually made a few videos, and let 00:04:55.190 --> 00:04:58.670 align:middle line:84% me play them because you can see it in action 00:04:58.670 --> 00:05:00.425 align:middle line:90% and see how it works. 00:05:00.425 --> 00:05:04.720 align:middle line:90% 00:05:04.720 --> 00:05:08.602 align:middle line:84% This is a real time capture, so this is the top of the box. 00:05:08.602 --> 00:05:09.310 align:middle line:90% You're zoomed in. 00:05:09.310 --> 00:05:12.610 align:middle line:84% You're looking at my finger pressing it to it. 00:05:12.610 --> 00:05:15.520 align:middle line:84% On the right is what the camera sees. 00:05:15.520 --> 00:05:18.130 align:middle line:84% This is looking at the bottom of the gel 00:05:18.130 --> 00:05:19.960 align:middle line:84% through glass, underneath the bottom. 00:05:19.960 --> 00:05:22.240 align:middle line:84% We have three colored lights around it, red, 00:05:22.240 --> 00:05:24.580 align:middle line:84% green, and blue, at different directions. 00:05:24.580 --> 00:05:29.140 align:middle line:84% And if you imagine what each of the individual color channels 00:05:29.140 --> 00:05:32.405 align:middle line:84% looks like in this image, if you just look at the red channel, 00:05:32.405 --> 00:05:34.030 align:middle line:84% it looks like a shaded image with light 00:05:34.030 --> 00:05:35.447 align:middle line:90% coming from this direction. 00:05:35.447 --> 00:05:37.030 align:middle line:84% If you just look at the green channel, 00:05:37.030 --> 00:05:39.790 align:middle line:84% the shaded image with light coming in that direction, same 00:05:39.790 --> 00:05:41.230 align:middle line:84% for the blue channel, shaded image 00:05:41.230 --> 00:05:42.910 align:middle line:84% with light coming in this direction. 00:05:42.910 --> 00:05:45.820 align:middle line:84% So by using three different colored lights and an RGB 00:05:45.820 --> 00:05:50.605 align:middle line:84% camera, we can have three separate shaded images. 00:05:50.605 --> 00:05:53.290 align:middle line:84% It's gray-scale images, but it's three different light 00:05:53.290 --> 00:05:54.460 align:middle line:90% directions. 00:05:54.460 --> 00:05:58.450 align:middle line:84% And it turns out that that is enough to constrain 00:05:58.450 --> 00:06:03.167 align:middle line:84% the surface normal on the shape of the top of the objects. 00:06:03.167 --> 00:06:04.750 align:middle line:84% I can talk a little bit about the math 00:06:04.750 --> 00:06:07.810 align:middle line:84% after this, but just to give you an appreciation for what 00:06:07.810 --> 00:06:09.550 align:middle line:90% the demo is like-- 00:06:09.550 --> 00:06:11.050 align:middle line:84% so this is what the camera is seeing 00:06:11.050 --> 00:06:13.030 align:middle line:84% as I move my finger around, but we 00:06:13.030 --> 00:06:16.240 align:middle line:84% can do a reconstruction of the geometry in real time. 00:06:16.240 --> 00:06:19.360 align:middle line:84% So I'm spinning the viewpoint, and this is all 00:06:19.360 --> 00:06:25.570 align:middle line:84% captured geometry and another camera feed just 00:06:25.570 --> 00:06:26.860 align:middle line:90% to see the interaction. 00:06:26.860 --> 00:06:29.800 align:middle line:84% This is what looks like when you brush your teeth. 00:06:29.800 --> 00:06:32.260 align:middle line:84% And this is the reconstructed geometry 00:06:32.260 --> 00:06:36.260 align:middle line:84% of the toothbrush and its bristles. 00:06:36.260 --> 00:06:39.170 align:middle line:84% And so this is the demo that we did at Siggraph, 00:06:39.170 --> 00:06:42.050 align:middle line:84% and also I just did it at [? ITCP ?] last week. 00:06:42.050 --> 00:06:45.440 align:middle line:84% And you can see the Oreo, interesting properties. 00:06:45.440 --> 00:06:47.510 align:middle line:90% It's high resolution. 00:06:47.510 --> 00:06:52.970 align:middle line:84% It's interactive, so you can get dynamic 3D information. 00:06:52.970 --> 00:06:54.740 align:middle line:90% Now this is after the Oreo. 00:06:54.740 --> 00:06:57.050 align:middle line:84% I noticed there were crumbs on the sensor, 00:06:57.050 --> 00:06:59.150 align:middle line:84% so you can see the geometry of the crumbs 00:06:59.150 --> 00:07:01.820 align:middle line:84% as they roll underneath my finger. 00:07:01.820 --> 00:07:04.580 align:middle line:90% And this is Ted Adelson's pulse. 00:07:04.580 --> 00:07:06.860 align:middle line:84% He's put his wrist down on the sensor, 00:07:06.860 --> 00:07:10.400 align:middle line:84% and he's reconstructed all of his skin texture and also 00:07:10.400 --> 00:07:12.320 align:middle line:90% the pulse that's moving. 00:07:12.320 --> 00:07:16.670 align:middle line:84% And finally just for fun, the surface of the bill, and this 00:07:16.670 --> 00:07:18.860 align:middle line:90% is all done in real time. 00:07:18.860 --> 00:07:25.795 align:middle line:84% I have to press it with a flat plate, and you can see-- 00:07:25.795 --> 00:07:27.920 align:middle line:84% maybe not those of you who are looking at an angle, 00:07:27.920 --> 00:07:29.587 align:middle line:84% but if you're looking at it straight on, 00:07:29.587 --> 00:07:31.880 align:middle line:84% you can actually see the raised printing. 00:07:31.880 --> 00:07:35.780 align:middle line:84% So it's got a lot of different properties. 00:07:35.780 --> 00:07:38.780 align:middle line:84% I said, real time, high resolution, 00:07:38.780 --> 00:07:41.132 align:middle line:84% but it's not a traditional 3D scanner 00:07:41.132 --> 00:07:43.340 align:middle line:84% because you're not going to be able to scan your face 00:07:43.340 --> 00:07:45.000 align:middle line:84% or something with a lot of depth, 00:07:45.000 --> 00:07:51.260 align:middle line:84% can't imagine pressing your face down to the gel to get full 3D. 00:07:51.260 --> 00:07:52.970 align:middle line:84% We don't claim it's a 3D scanner anyway. 00:07:52.970 --> 00:07:54.410 align:middle line:84% It's really [? 2.5 ?] [INAUDIBLE] 00:07:54.410 --> 00:07:57.050 align:middle line:90% a type map in real time. 00:07:57.050 --> 00:07:59.480 align:middle line:84% So let me go back to the presentation. 00:07:59.480 --> 00:08:03.060 align:middle line:90% 00:08:03.060 --> 00:08:04.470 align:middle line:90% How does it work? 00:08:04.470 --> 00:08:07.110 align:middle line:84% Well, why do you need three different light directions? 00:08:07.110 --> 00:08:10.620 align:middle line:84% Why don't you just have one light 00:08:10.620 --> 00:08:13.050 align:middle line:84% and try to reconstruct the shape with one light? 00:08:13.050 --> 00:08:15.060 align:middle line:84% Well, that's called shape from shading. 00:08:15.060 --> 00:08:17.370 align:middle line:84% And in computer vision that's considered 00:08:17.370 --> 00:08:20.070 align:middle line:84% one of the traditional hard problems. 00:08:20.070 --> 00:08:22.440 align:middle line:84% So we don't want to have to attack a hard problem, 00:08:22.440 --> 00:08:26.670 align:middle line:84% especially if we want this to happen in real time. 00:08:26.670 --> 00:08:30.210 align:middle line:84% We use photometric stereo, which is actually 00:08:30.210 --> 00:08:34.020 align:middle line:84% a very old technique too, first proposed by Woodham in 1980. 00:08:34.020 --> 00:08:35.909 align:middle line:84% And it says, if you have three lights-- 00:08:35.909 --> 00:08:38.669 align:middle line:84% and this is a Lambertian sphere lit from three 00:08:38.669 --> 00:08:39.690 align:middle line:90% different directions. 00:08:39.690 --> 00:08:42.649 align:middle line:84% So you can see the lights are in each of the color channels red, 00:08:42.649 --> 00:08:45.210 align:middle line:90% green, and blue. 00:08:45.210 --> 00:08:47.790 align:middle line:84% They provide constraints on the surface [? level. ?] 00:08:47.790 --> 00:08:49.440 align:middle line:90% We'll talk more about that. 00:08:49.440 --> 00:08:52.020 align:middle line:90% 00:08:52.020 --> 00:08:54.570 align:middle line:84% Here's the three shaded views, just so you can see. 00:08:54.570 --> 00:08:56.280 align:middle line:84% The Oreo's pressed into the sensor. 00:08:56.280 --> 00:08:59.070 align:middle line:84% This is an actual picture that I took 00:08:59.070 --> 00:09:01.900 align:middle line:84% with it pressed in the sensor, red channel, the green channel, 00:09:01.900 --> 00:09:02.820 align:middle line:90% and the blue channel. 00:09:02.820 --> 00:09:04.195 align:middle line:84% And you can see the lights coming 00:09:04.195 --> 00:09:06.510 align:middle line:90% from different directions. 00:09:06.510 --> 00:09:08.530 align:middle line:84% AUDIENCE: Why don't you have broad sources? 00:09:08.530 --> 00:09:08.790 align:middle line:90% They're not-- 00:09:08.790 --> 00:09:10.165 align:middle line:84% MICAH KIMO JOHNSON: Yeah, they're 00:09:10.165 --> 00:09:12.540 align:middle line:84% actually sort of little area light sources, 00:09:12.540 --> 00:09:16.890 align:middle line:84% and we found that we get the best looking results in terms 00:09:16.890 --> 00:09:20.040 align:middle line:84% of detail by using specular paints, which 00:09:20.040 --> 00:09:23.992 align:middle line:84% have a lot of contrast, with area light sources. 00:09:23.992 --> 00:09:25.950 align:middle line:84% So there's a lot of things you can fiddle with. 00:09:25.950 --> 00:09:28.920 align:middle line:84% You could go to a point-light sources with a Lambertian 00:09:28.920 --> 00:09:32.820 align:middle line:84% or a very diffuse paint, and that's 00:09:32.820 --> 00:09:35.850 align:middle line:84% better at reconstructing a lot of depth. 00:09:35.850 --> 00:09:39.510 align:middle line:84% But to get fingerprints and details 00:09:39.510 --> 00:09:41.790 align:middle line:84% on the surface of a bill, we found 00:09:41.790 --> 00:09:47.820 align:middle line:84% that it works better to have glossy paint and broad lights. 00:09:47.820 --> 00:09:51.030 align:middle line:84% But, yeah, all sorts of things you can fiddle. 00:09:51.030 --> 00:09:55.173 align:middle line:90% 00:09:55.173 --> 00:09:57.340 align:middle line:84% Actually, let me just go into a little bit of detail 00:09:57.340 --> 00:09:59.007 align:middle line:84% why you need three lights just because I 00:09:59.007 --> 00:10:01.510 align:middle line:90% think it's interesting. 00:10:01.510 --> 00:10:05.020 align:middle line:84% So again, as I said, if you'd only had one light, 00:10:05.020 --> 00:10:07.080 align:middle line:84% you'd have to solve shape from shading problem. 00:10:07.080 --> 00:10:07.960 align:middle line:90% Why is this hard? 00:10:07.960 --> 00:10:11.020 align:middle line:84% So here's a shaded sphere with a single light direction. 00:10:11.020 --> 00:10:14.300 align:middle line:84% This is a Lambertian, so it's a diffuse sphere. 00:10:14.300 --> 00:10:21.080 align:middle line:84% And let's say you know that the intensity value 00:10:21.080 --> 00:10:24.470 align:middle line:90% of a certain pixel is 0.9. 00:10:24.470 --> 00:10:27.350 align:middle line:84% Well, here I find a [? iso ?] [? platonic ?] sphere, 00:10:27.350 --> 00:10:29.420 align:middle line:90% so [? 0.99 ?] here. 00:10:29.420 --> 00:10:30.800 align:middle line:84% But in fact, the orientation that 00:10:30.800 --> 00:10:37.065 align:middle line:84% corresponds to 0.9 [INAUDIBLE] because all of these 00:10:37.065 --> 00:10:38.440 align:middle line:84% points having a [? normal ?] that 00:10:38.440 --> 00:10:40.870 align:middle line:84% makes the same angle of a light direction. 00:10:40.870 --> 00:10:45.220 align:middle line:84% And that's how a sphere shaded in a diffuse kit. 00:10:45.220 --> 00:10:48.867 align:middle line:84% So just knowing the shading at a particular pixel that is 0.9, 00:10:48.867 --> 00:10:50.950 align:middle line:84% it doesn't tell you anything about the orientation 00:10:50.950 --> 00:10:53.110 align:middle line:90% of the surface underneath. 00:10:53.110 --> 00:10:56.200 align:middle line:84% But by having three light directions, 00:10:56.200 --> 00:11:01.690 align:middle line:84% if it's 0.9 in green, 0.8 in red, and 0.7 in blue, 00:11:01.690 --> 00:11:04.450 align:middle line:84% or something, you can pinpoint-- we have the intersection 00:11:04.450 --> 00:11:07.860 align:middle line:84% of these, these types of [? ice ?] [? coats. ?] And you 00:11:07.860 --> 00:11:10.360 align:middle line:84% can pinpoint the location on these [INAUDIBLE] three lights. 00:11:10.360 --> 00:11:13.660 align:middle line:84% So it's really pretty much that simple, 00:11:13.660 --> 00:11:16.930 align:middle line:84% where you look at the colors in each channel, 00:11:16.930 --> 00:11:20.440 align:middle line:84% and you build a lookup table through calibration. 00:11:20.440 --> 00:11:25.630 align:middle line:84% So we calibrate by pressing into it this grid of spheres. 00:11:25.630 --> 00:11:28.103 align:middle line:84% We know the size, and we know the geometry 00:11:28.103 --> 00:11:29.020 align:middle line:90% because it's a sphere. 00:11:29.020 --> 00:11:30.160 align:middle line:90% It's very simple. 00:11:30.160 --> 00:11:32.170 align:middle line:84% You press it in, and it looks like this. 00:11:32.170 --> 00:11:36.250 align:middle line:84% And already you can see that color denotes orientation. 00:11:36.250 --> 00:11:39.070 align:middle line:84% If you see a pure green pixel, it 00:11:39.070 --> 00:11:40.990 align:middle line:84% means the orientation is something like this. 00:11:40.990 --> 00:11:42.532 align:middle line:84% If you see a pure red pixel, it means 00:11:42.532 --> 00:11:44.320 align:middle line:84% your orientation is down like that, 00:11:44.320 --> 00:11:45.953 align:middle line:90% and blue is over like this. 00:11:45.953 --> 00:11:47.620 align:middle line:84% And then if you see a mixture of colors, 00:11:47.620 --> 00:11:51.100 align:middle line:84% it means it's some mixture in between those orientations. 00:11:51.100 --> 00:11:53.320 align:middle line:84% So you can build a mapping from color 00:11:53.320 --> 00:11:56.950 align:middle line:84% to surface normal, so it's a 3D lookup table. 00:11:56.950 --> 00:12:02.440 align:middle line:84% We have red, green, and blue indices, and each of them 00:12:02.440 --> 00:12:05.050 align:middle line:84% you can get the gradient or the surface normal. 00:12:05.050 --> 00:12:09.310 align:middle line:84% So again, if we see color, 10 in red, 200 in green, 00:12:09.310 --> 00:12:11.770 align:middle line:84% and 3 in blue, we go to the lookup table, 00:12:11.770 --> 00:12:16.000 align:middle line:84% and we get out that gradient, something like that. 00:12:16.000 --> 00:12:18.900 align:middle line:84% So you get a different color, you get a different gradient. 00:12:18.900 --> 00:12:23.040 align:middle line:84% So then we solved Poisson equation 00:12:23.040 --> 00:12:25.500 align:middle line:84% to reconstruct the surface from its gradients. 00:12:25.500 --> 00:12:27.510 align:middle line:90% Ramesh mentioned that before. 00:12:27.510 --> 00:12:32.580 align:middle line:84% It's fairly common and used in a lot of different problems. 00:12:32.580 --> 00:12:33.810 align:middle line:90% But here are some results. 00:12:33.810 --> 00:12:39.810 align:middle line:84% So these are the results from the [? CBC ?] paper. 00:12:39.810 --> 00:12:42.870 align:middle line:84% This would be really challenging to use 00:12:42.870 --> 00:12:46.950 align:middle line:84% laser scanning or traditional photometric stereo. 00:12:46.950 --> 00:12:50.700 align:middle line:84% But when you press it in, the skin 00:12:50.700 --> 00:12:54.150 align:middle line:84% of the gel potentially paints [? sideways. ?] 00:12:54.150 --> 00:12:56.820 align:middle line:84% Everything you press into it looks like it's been painting. 00:12:56.820 --> 00:13:00.240 align:middle line:84% So we can reconstruct the geometry, and you get-- 00:13:00.240 --> 00:13:04.710 align:middle line:84% this is head-on view and a view from a different angle, 00:13:04.710 --> 00:13:07.380 align:middle line:84% and it's very direct and straightforward. 00:13:07.380 --> 00:13:11.220 align:middle line:84% So again, here are these objects that I showed before. 00:13:11.220 --> 00:13:14.820 align:middle line:84% And you can see we've captured a lot of the details. 00:13:14.820 --> 00:13:16.500 align:middle line:84% Now you don't get color information 00:13:16.500 --> 00:13:17.850 align:middle line:84% because you're essentially painting it 00:13:17.850 --> 00:13:19.142 align:middle line:90% when you press it into the gel. 00:13:19.142 --> 00:13:22.860 align:middle line:84% But in a lot of cases for graphics, 00:13:22.860 --> 00:13:27.330 align:middle line:84% just having bump maps, having the texture is useful. 00:13:27.330 --> 00:13:32.310 align:middle line:84% And finally, the 20 [? probably-- ?] 00:13:32.310 --> 00:13:35.630 align:middle line:84% so for the [? CVCR ?] paper, I did a 20. 00:13:35.630 --> 00:13:38.040 align:middle line:84% I stepped it up a bit for the video and did a hundred, 00:13:38.040 --> 00:13:42.150 align:middle line:90% but they both work pretty well. 00:13:42.150 --> 00:13:44.190 align:middle line:84% So that's the height map that we've measured. 00:13:44.190 --> 00:13:47.520 align:middle line:84% And now you can render, and you can change the viewpoint. 00:13:47.520 --> 00:13:51.750 align:middle line:84% And you can see that we've captured quite a bit of detail. 00:13:51.750 --> 00:13:55.420 align:middle line:84% And that's just the security strip [INAUDIBLE].. 00:13:55.420 --> 00:13:57.420 align:middle line:84% AUDIENCE: Going to get into counterfeiting then? 00:13:57.420 --> 00:13:58.020 align:middle line:90% MICAH KIMO JOHNSON: Yeah. 00:13:58.020 --> 00:13:58.380 align:middle line:90% yeah. 00:13:58.380 --> 00:14:00.297 align:middle line:84% You can actually sending this to a 3D printer. 00:14:00.297 --> 00:14:08.300 align:middle line:90% 00:14:08.300 --> 00:14:13.520 align:middle line:84% So recently, we showed this, like I 00:14:13.520 --> 00:14:15.170 align:middle line:84% said at Siggraph Emerging Technologies, 00:14:15.170 --> 00:14:19.310 align:middle line:84% and we took sort of a graphics slant showing 00:14:19.310 --> 00:14:21.200 align:middle line:90% how you can measure surfaces. 00:14:21.200 --> 00:14:26.750 align:middle line:84% But this could be used for a lot more. 00:14:26.750 --> 00:14:33.140 align:middle line:84% So we just made a second video, and I talk a lot in this one, 00:14:33.140 --> 00:14:35.735 align:middle line:84% but let me just fast forward to the interesting ideas. 00:14:35.735 --> 00:14:38.710 align:middle line:90% 00:14:38.710 --> 00:14:39.210 align:middle line:90% OK. 00:14:39.210 --> 00:14:44.400 align:middle line:84% So in this video, we're going to look at how can we 00:14:44.400 --> 00:14:49.080 align:middle line:84% change the properties of the gel to measure 00:14:49.080 --> 00:14:51.122 align:middle line:84% different things, perhaps beyond the limits 00:14:51.122 --> 00:14:52.080 align:middle line:90% of the original system. 00:14:52.080 --> 00:14:56.790 align:middle line:84% So how can we get even higher spatial resolution? 00:14:56.790 --> 00:14:59.430 align:middle line:84% How can we change the sensitivity of the system? 00:14:59.430 --> 00:15:02.402 align:middle line:84% And can it be made into a smaller device? 00:15:02.402 --> 00:15:04.860 align:middle line:84% Because everyone asked us when they saw this big box that I 00:15:04.860 --> 00:15:07.230 align:middle line:84% had, well, you know, they said, that seems 00:15:07.230 --> 00:15:08.820 align:middle line:90% really bulky and cumbersome. 00:15:08.820 --> 00:15:10.230 align:middle line:90% Does it have to be that big? 00:15:10.230 --> 00:15:15.040 align:middle line:84% So we've made a smaller version, but let me step 00:15:15.040 --> 00:15:16.510 align:middle line:90% through some of these results. 00:15:16.510 --> 00:15:22.000 align:middle line:84% So spatial resolution, this is an oregano leaf 00:15:22.000 --> 00:15:24.790 align:middle line:84% pressed into a very thin piece of this sensor. 00:15:24.790 --> 00:15:27.520 align:middle line:84% So it looks like the oregano leaf has been painted. 00:15:27.520 --> 00:15:31.830 align:middle line:84% And you can see a lot of detail with your naked eye like this, 00:15:31.830 --> 00:15:33.580 align:middle line:84% but then you can put it under a microscope 00:15:33.580 --> 00:15:36.660 align:middle line:90% too, and zoom way in. 00:15:36.660 --> 00:15:38.410 align:middle line:84% And you can see a lot of these structures, 00:15:38.410 --> 00:15:40.327 align:middle line:84% and I don't know the technical terms for them. 00:15:40.327 --> 00:15:44.800 align:middle line:84% But you've got these little pieces here, these hairs. 00:15:44.800 --> 00:15:48.460 align:middle line:84% We estimate that, I believe, some of these structures 00:15:48.460 --> 00:15:49.660 align:middle line:90% are 100 microns. 00:15:49.660 --> 00:15:51.220 align:middle line:84% And then at the tip of the hairs, 00:15:51.220 --> 00:15:54.310 align:middle line:84% that's perhaps on the order of just a few microns. 00:15:54.310 --> 00:15:58.330 align:middle line:84% So you can really see a lot of detail 00:15:58.330 --> 00:16:00.190 align:middle line:84% just by pressing this gel into it. 00:16:00.190 --> 00:16:01.640 align:middle line:90% AUDIENCE: Wait. 00:16:01.640 --> 00:16:03.490 align:middle line:84% Is this the 3D reconstruction or just-- 00:16:03.490 --> 00:16:05.657 align:middle line:84% MICAH KIMO JOHNSON: This is not a 3D reconstruction, 00:16:05.657 --> 00:16:08.050 align:middle line:84% but the same principle could apply. 00:16:08.050 --> 00:16:09.700 align:middle line:84% It's just a different piece of gel, 00:16:09.700 --> 00:16:11.260 align:middle line:90% and we just took a picture. 00:16:11.260 --> 00:16:13.450 align:middle line:84% If we put the three lights in and calibrated, 00:16:13.450 --> 00:16:15.880 align:middle line:84% we could get the 3D geometry out of this. 00:16:15.880 --> 00:16:18.610 align:middle line:84% Calibration might be a bit challenging and getting 00:16:18.610 --> 00:16:21.400 align:middle line:84% the lights configured, but in principle, this 00:16:21.400 --> 00:16:23.470 align:middle line:84% is no different from what we did before. 00:16:23.470 --> 00:16:25.810 align:middle line:90% So then we look at sensitivity. 00:16:25.810 --> 00:16:30.403 align:middle line:90% 00:16:30.403 --> 00:16:32.320 align:middle line:84% A lot of these are just Ted Adelson's results. 00:16:32.320 --> 00:16:35.710 align:middle line:84% He's the one who came up with the idea of the gel and this. 00:16:35.710 --> 00:16:38.170 align:middle line:84% And he's been fiddling with these things in his basement. 00:16:38.170 --> 00:16:39.880 align:middle line:84% I did the 3D reconstruction stuff. 00:16:39.880 --> 00:16:42.400 align:middle line:84% But I didn't even know he was thinking 00:16:42.400 --> 00:16:45.460 align:middle line:84% about all these different ideas, and he sent me these videos. 00:16:45.460 --> 00:16:52.390 align:middle line:84% So this video is his car driving over a piece of this sensor. 00:16:52.390 --> 00:16:56.355 align:middle line:84% So you think, how did he do this? 00:16:56.355 --> 00:16:58.480 align:middle line:84% Well, first of all, why he did it, so I don't know. 00:16:58.480 --> 00:17:01.780 align:middle line:90% But the idea is that-- 00:17:01.780 --> 00:17:05.890 align:middle line:84% the gel that a lot of people have played with in our demo 00:17:05.890 --> 00:17:07.089 align:middle line:90% is soft to touch. 00:17:07.089 --> 00:17:09.724 align:middle line:84% So you touch it, deforms, and you can see your fingerprints 00:17:09.724 --> 00:17:10.599 align:middle line:90% and things like this. 00:17:10.599 --> 00:17:15.310 align:middle line:84% But you can get these rubbers in a variety of strengths, 00:17:15.310 --> 00:17:16.300 align:middle line:90% variety of elasticity. 00:17:16.300 --> 00:17:21.160 align:middle line:84% So he went out and got one that was as hard as a tire itself. 00:17:21.160 --> 00:17:22.720 align:middle line:84% And then he bought one of these ramps 00:17:22.720 --> 00:17:24.130 align:middle line:84% that you used to change your oil, 00:17:24.130 --> 00:17:26.440 align:middle line:90% so you can drive a car up on it. 00:17:26.440 --> 00:17:29.110 align:middle line:84% He cut a hole in the top, put some of this gel down, 00:17:29.110 --> 00:17:31.420 align:middle line:84% put a mirror underneath that had a camera, 00:17:31.420 --> 00:17:32.980 align:middle line:90% and then drove his car over it. 00:17:32.980 --> 00:17:37.630 align:middle line:84% You can see these are some pebbles stuck in the tire. 00:17:37.630 --> 00:17:39.640 align:middle line:90% This is the view of the tire. 00:17:39.640 --> 00:17:41.830 align:middle line:84% Now this is not reconstructed 3D geometry, 00:17:41.830 --> 00:17:44.090 align:middle line:90% but again, it could be that. 00:17:44.090 --> 00:17:47.860 align:middle line:84% You could actually measure tread wear or something like that. 00:17:47.860 --> 00:17:52.690 align:middle line:84% It just shows that the system can handle 2,800 pounds of car 00:17:52.690 --> 00:17:54.190 align:middle line:90% on top of it. 00:17:54.190 --> 00:18:02.420 align:middle line:84% But on the other side, this is him pressing, 00:18:02.420 --> 00:18:05.900 align:middle line:90% just poking at with a hair. 00:18:05.900 --> 00:18:11.560 align:middle line:84% So this is a very soft piece of gel, 00:18:11.560 --> 00:18:14.080 align:middle line:84% and he's just touching it with an end of a hair. 00:18:14.080 --> 00:18:16.750 align:middle line:84% Or on the other hand, this is some soap bubbles 00:18:16.750 --> 00:18:19.180 align:middle line:90% viewed through clear glass. 00:18:19.180 --> 00:18:22.270 align:middle line:84% So this container was clear glass and soap bubbles inside. 00:18:22.270 --> 00:18:25.360 align:middle line:84% But then he's put sensor material on the other side, 00:18:25.360 --> 00:18:29.440 align:middle line:84% and you can see that just the pressure of the soap bubbles 00:18:29.440 --> 00:18:31.210 align:middle line:90% is enough to deform the gel. 00:18:31.210 --> 00:18:35.868 align:middle line:84% So on the one hand, we had a car on top of it. 00:18:35.868 --> 00:18:37.660 align:middle line:84% And the pressure from the car, to start it, 00:18:37.660 --> 00:18:39.220 align:middle line:90% we could get geometry there. 00:18:39.220 --> 00:18:42.343 align:middle line:84% On the other extreme, we have soap bubbles, 00:18:42.343 --> 00:18:44.260 align:middle line:84% and you can see the geometry, and you can even 00:18:44.260 --> 00:18:46.990 align:middle line:84% track how they form and how they change. 00:18:46.990 --> 00:18:50.650 align:middle line:84% So there's a wide range of sensitivity 00:18:50.650 --> 00:18:53.470 align:middle line:90% to do anything in between. 00:18:53.470 --> 00:18:57.760 align:middle line:90% This is somewhat disturbing. 00:18:57.760 --> 00:19:00.305 align:middle line:90% So in talking to-- 00:19:00.305 --> 00:19:02.680 align:middle line:84% and he sends it with this in his email, and I open it up, 00:19:02.680 --> 00:19:03.410 align:middle line:90% and it plays. 00:19:03.410 --> 00:19:05.680 align:middle line:90% It's like [? disturbing. ?] 00:19:05.680 --> 00:19:07.300 align:middle line:84% And talking to technology licensing, 00:19:07.300 --> 00:19:10.420 align:middle line:84% they thought, wouldn't this be interesting to do 00:19:10.420 --> 00:19:11.890 align:middle line:90% medical imaging? 00:19:11.890 --> 00:19:14.980 align:middle line:84% So there's some exams, like prostate exams that 00:19:14.980 --> 00:19:16.990 align:middle line:84% are done by a doctor, putting their finger in, 00:19:16.990 --> 00:19:20.200 align:middle line:90% feeling, so the touch-based. 00:19:20.200 --> 00:19:22.120 align:middle line:84% And then they write down what they feel, 00:19:22.120 --> 00:19:23.650 align:middle line:90% but there's no record of that. 00:19:23.650 --> 00:19:27.370 align:middle line:84% There's no image that they take, nothing like that. 00:19:27.370 --> 00:19:28.990 align:middle line:84% This is a touch-based sensor that 00:19:28.990 --> 00:19:31.540 align:middle line:84% can capture an image of what it touches. 00:19:31.540 --> 00:19:34.260 align:middle line:84% So in theory, you can build it small enough, 00:19:34.260 --> 00:19:38.950 align:middle line:84% capture the geometry of internal tissues. 00:19:38.950 --> 00:19:41.650 align:middle line:84% But rather than make something that we would put internally, 00:19:41.650 --> 00:19:44.940 align:middle line:84% Ted just licked it to show internal tissues was 00:19:44.940 --> 00:19:45.565 align:middle line:90% being measured. 00:19:45.565 --> 00:19:50.700 align:middle line:90% 00:19:50.700 --> 00:19:51.570 align:middle line:90% Let's see. 00:19:51.570 --> 00:19:53.805 align:middle line:90% We've got, oh, yes, worms. 00:19:53.805 --> 00:19:56.910 align:middle line:90% 00:19:56.910 --> 00:19:59.130 align:middle line:84% These are worms on the surface of the sensor. 00:19:59.130 --> 00:20:02.340 align:middle line:84% And looking from the top, you can't really 00:20:02.340 --> 00:20:04.860 align:middle line:84% tell how the worms move very easily because they're 00:20:04.860 --> 00:20:05.910 align:middle line:90% translucent. 00:20:05.910 --> 00:20:11.550 align:middle line:84% You can't see the details, so they're not 00:20:11.550 --> 00:20:12.800 align:middle line:90% so obvious what they're doing. 00:20:12.800 --> 00:20:15.770 align:middle line:84% But at the same time, he was recording underneath. 00:20:15.770 --> 00:20:17.480 align:middle line:84% So this is the view from the sensor. 00:20:17.480 --> 00:20:23.730 align:middle line:84% And you can see that as the worm expands and contracts, 00:20:23.730 --> 00:20:26.700 align:middle line:84% you see these protrusions here, these structures that stick out 00:20:26.700 --> 00:20:28.080 align:middle line:90% and grip. 00:20:28.080 --> 00:20:30.270 align:middle line:84% And I learned they're called [? CT. ?] 00:20:30.270 --> 00:20:34.650 align:middle line:84% And perhaps people who are interested in saying worms 00:20:34.650 --> 00:20:38.130 align:middle line:84% might be interested in seeing or measuring 00:20:38.130 --> 00:20:40.680 align:middle line:84% how they expand and contract and how these little structures 00:20:40.680 --> 00:20:41.910 align:middle line:90% come out. 00:20:41.910 --> 00:20:45.720 align:middle line:84% So we can just see it visually, much clearer because we've 00:20:45.720 --> 00:20:47.730 align:middle line:90% essentially painted the worm. 00:20:47.730 --> 00:20:50.220 align:middle line:84% And finally to address the question of can 00:20:50.220 --> 00:20:52.770 align:middle line:90% we make it smaller? 00:20:52.770 --> 00:20:55.260 align:middle line:90% Ted made a compact version. 00:20:55.260 --> 00:21:00.780 align:middle line:84% And let me explain this one a little bit. 00:21:00.780 --> 00:21:06.960 align:middle line:84% This is a dental camera, so it's somewhat like a toothbrush. 00:21:06.960 --> 00:21:09.660 align:middle line:84% But inside here is a camera and a light. 00:21:09.660 --> 00:21:10.860 align:middle line:90% It's very small. 00:21:10.860 --> 00:21:13.980 align:middle line:84% And so on top he built like a lens and piece 00:21:13.980 --> 00:21:15.872 align:middle line:90% of sensor material. 00:21:15.872 --> 00:21:16.580 align:middle line:90% You can see that. 00:21:16.580 --> 00:21:18.330 align:middle line:84% It has a little bit of memory in this one. 00:21:18.330 --> 00:21:22.017 align:middle line:84% But when he touches it, it's the same principle 00:21:22.017 --> 00:21:23.100 align:middle line:90% of what we've done before. 00:21:23.100 --> 00:21:24.642 align:middle line:84% It's got some of the sensor material. 00:21:24.642 --> 00:21:27.150 align:middle line:84% But now it's in this handheld device, 00:21:27.150 --> 00:21:31.020 align:middle line:84% and it could be made even smaller if we just engineered 00:21:31.020 --> 00:21:32.340 align:middle line:90% smaller light, smaller cameras. 00:21:32.340 --> 00:21:33.960 align:middle line:84% This was just something off the shelf that we bought, 00:21:33.960 --> 00:21:35.850 align:middle line:84% a dental camera, so it's pretty small. 00:21:35.850 --> 00:21:39.870 align:middle line:84% And we can show that you just touch it into things, 00:21:39.870 --> 00:21:42.540 align:middle line:84% and you can see the texture in geometry. 00:21:42.540 --> 00:21:45.060 align:middle line:84% And with three lights, we could do our same trick 00:21:45.060 --> 00:21:48.015 align:middle line:84% of photometric stereo and actually measuring the geometry 00:21:48.015 --> 00:21:49.710 align:middle line:90% and reconstructing it. 00:21:49.710 --> 00:21:54.960 align:middle line:84% But this one is kind of fun to play with. 00:21:54.960 --> 00:21:58.170 align:middle line:84% You can see this might be useful for skin texture. 00:21:58.170 --> 00:22:01.410 align:middle line:84% So one of the last remaining challenges 00:22:01.410 --> 00:22:04.860 align:middle line:84% of doing realistic human rendering 00:22:04.860 --> 00:22:07.770 align:middle line:84% is getting this microstructure of skin, the wrinkles, 00:22:07.770 --> 00:22:10.530 align:middle line:84% the pores, all the things like that. 00:22:10.530 --> 00:22:17.713 align:middle line:84% And people have built some very expensive domes 00:22:17.713 --> 00:22:19.380 align:middle line:84% to try to measure a lot of the geometry, 00:22:19.380 --> 00:22:21.930 align:middle line:84% but they can't get down to this level of detail. 00:22:21.930 --> 00:22:25.203 align:middle line:84% We can get it locally, but we can't get the low resolution. 00:22:25.203 --> 00:22:27.120 align:middle line:84% We can't get the actual structure of the face, 00:22:27.120 --> 00:22:29.910 align:middle line:84% so maybe combining these techniques 00:22:29.910 --> 00:22:33.610 align:middle line:84% could yield something interesting. 00:22:33.610 --> 00:22:36.280 align:middle line:90% So here's stubble. 00:22:36.280 --> 00:22:40.570 align:middle line:84% You can see the stubble actually has quite a bit of geometry, 00:22:40.570 --> 00:22:43.945 align:middle line:84% more than you might think, and his hair. 00:22:43.945 --> 00:22:47.930 align:middle line:90% 00:22:47.930 --> 00:22:51.538 align:middle line:84% That's basically the touch-sensor system. 00:22:51.538 --> 00:22:52.830 align:middle line:90% Does anyone have any questions? 00:22:52.830 --> 00:22:56.538 align:middle line:90% 00:22:56.538 --> 00:22:58.080 align:middle line:84% AUDIENCE: Yeah, I do have a question. 00:22:58.080 --> 00:23:01.450 align:middle line:84% So if you use that little [INAUDIBLE] 00:23:01.450 --> 00:23:04.720 align:middle line:84% table to get some information and get 00:23:04.720 --> 00:23:06.880 align:middle line:84% the direction in the wrong findings, 00:23:06.880 --> 00:23:10.690 align:middle line:84% but you may have several problems separately 00:23:10.690 --> 00:23:13.730 align:middle line:84% or from just a different path, and you need 00:23:13.730 --> 00:23:16.718 align:middle line:90% to straighten that [INAUDIBLE] 00:23:16.718 --> 00:23:18.010 align:middle line:90% MICAH KIMO JOHNSON: Right, so-- 00:23:18.010 --> 00:23:18.310 align:middle line:90% [INTERPOSING VOICES] 00:23:18.310 --> 00:23:19.935 align:middle line:84% AUDIENCE: How would you get that right? 00:23:19.935 --> 00:23:20.920 align:middle line:90% You can say, OK. 00:23:20.920 --> 00:23:24.568 align:middle line:84% How would you distinguish this [INAUDIBLE] 00:23:24.568 --> 00:23:26.610 align:middle line:84% MICAH KIMO JOHNSON: You're relying on the ability 00:23:26.610 --> 00:23:29.760 align:middle line:84% to measure orientation accurately. 00:23:29.760 --> 00:23:33.450 align:middle line:84% So we're measuring orientation, the surface normal. 00:23:33.450 --> 00:23:37.890 align:middle line:84% And we've verified that for relatively 00:23:37.890 --> 00:23:40.830 align:middle line:84% shallow orientations, meaning orientations that basically 00:23:40.830 --> 00:23:42.690 align:middle line:84% face the camera we're pretty accurate. 00:23:42.690 --> 00:23:45.810 align:middle line:84% As you get steep orientations, we've become less accurate. 00:23:45.810 --> 00:23:48.473 align:middle line:84% And we could employ other techniques 00:23:48.473 --> 00:23:50.640 align:middle line:84% to have more accuracy there, like maybe more lights, 00:23:50.640 --> 00:23:51.750 align:middle line:90% more views. 00:23:51.750 --> 00:23:55.290 align:middle line:84% But in general, we're not getting steep normals accurate. 00:23:55.290 --> 00:23:59.760 align:middle line:84% So we reconstruct the depth from the normals, 00:23:59.760 --> 00:24:02.490 align:middle line:90% so there might be some bias. 00:24:02.490 --> 00:24:04.980 align:middle line:84% Let's say the surface had a dip like this. 00:24:04.980 --> 00:24:06.780 align:middle line:84% If you missestimate this normal, you 00:24:06.780 --> 00:24:09.750 align:middle line:84% might only reconstruct it like that, 00:24:09.750 --> 00:24:11.970 align:middle line:90% so you can get errors like this. 00:24:11.970 --> 00:24:17.550 align:middle line:84% But if you're mostly interested in high-resolution detail, 00:24:17.550 --> 00:24:19.200 align:middle line:90% this device is pretty good. 00:24:19.200 --> 00:24:22.680 align:middle line:84% If you need to get those low-resolution, 00:24:22.680 --> 00:24:25.980 align:middle line:84% depth estimates accurate, you might need more views 00:24:25.980 --> 00:24:27.870 align:middle line:84% or more lights than what we're using. 00:24:27.870 --> 00:24:30.690 align:middle line:90% 00:24:30.690 --> 00:24:35.730 align:middle line:84% AUDIENCE: What is the state-of-the-art in touch 00:24:35.730 --> 00:24:37.860 align:middle line:84% senses that you maybe drag over something that 00:24:37.860 --> 00:24:39.615 align:middle line:90% aren't camera-based? 00:24:39.615 --> 00:24:40.990 align:middle line:84% I mean is there anything in that? 00:24:40.990 --> 00:24:43.365 align:middle line:84% MICAH KIMO JOHNSON: Yeah, I think the state-of-the-art is 00:24:43.365 --> 00:24:45.720 align:middle line:84% 10 [? prolimens ?] for [INAUDIBLE] internal 00:24:45.720 --> 00:24:47.130 align:middle line:90% reflection. 00:24:47.130 --> 00:24:48.720 align:middle line:90% That's something which-- 00:24:48.720 --> 00:24:51.240 align:middle line:84% [? RAMESH RASKAR: ?] Defining [INAUDIBLE],, NYU? 00:24:51.240 --> 00:24:52.240 align:middle line:90% MICAH KIMO JOHNSON: NYU. 00:24:52.240 --> 00:24:54.975 align:middle line:84% But [INAUDIBLE] to it because he was on it. 00:24:54.975 --> 00:24:56.850 align:middle line:84% RAMESH RASKAR: Now you're talking about the-- 00:24:56.850 --> 00:24:58.110 align:middle line:84% MICAH KIMO JOHNSON: Oh, there's the Microsoft Surface, 00:24:58.110 --> 00:24:58.860 align:middle line:90% but then there's this new-- 00:24:58.860 --> 00:25:00.000 align:middle line:90% RAMESH RASKAR: No, no. 00:25:00.000 --> 00:25:01.912 align:middle line:84% Ken Karlin has a new force-based-- 00:25:01.912 --> 00:25:03.620 align:middle line:84% MICAH KIMO JOHNSON: Oh, it's force based. 00:25:03.620 --> 00:25:06.870 align:middle line:84% RAMESH RASKAR: --multi-touch sensor, which he has shown in-- 00:25:06.870 --> 00:25:07.940 align:middle line:90% AUDIENCE: Is it like gel? 00:25:07.940 --> 00:25:09.760 align:middle line:84% RAMESH RASKAR: Yeah, the gel one, and then 00:25:09.760 --> 00:25:11.760 align:middle line:84% the [? internal ?] reflection is purely optical. 00:25:11.760 --> 00:25:14.058 align:middle line:84% MICAH KIMO JOHNSON: I was going to ask about that. 00:25:14.058 --> 00:25:16.350 align:middle line:84% AUDIENCE: Is there anything where you directly measure? 00:25:16.350 --> 00:25:18.540 align:middle line:84% I can imagine a line of points that you 00:25:18.540 --> 00:25:21.150 align:middle line:84% can measure the height that they get to press, 00:25:21.150 --> 00:25:22.770 align:middle line:84% and you drag your finger across it. 00:25:22.770 --> 00:25:24.600 align:middle line:90% Is there anything like that? 00:25:24.600 --> 00:25:26.820 align:middle line:84% RAMESH RASKAR: How would the height change help you? 00:25:26.820 --> 00:25:28.320 align:middle line:84% AUDIENCE: Like where you're directly 00:25:28.320 --> 00:25:31.552 align:middle line:84% measuring the height instead of inferring it based on images. 00:25:31.552 --> 00:25:32.760 align:middle line:90% AUDIENCE: Oh, yeah, there is. 00:25:32.760 --> 00:25:33.843 align:middle line:90% The [? Soundscape ?] demo. 00:25:33.843 --> 00:25:39.280 align:middle line:90% 00:25:39.280 --> 00:25:44.250 align:middle line:84% MICAH KIMO JOHNSON: I think that that one [? Ken ?] 00:25:44.250 --> 00:25:48.780 align:middle line:84% [? Karlin's ?] one does measure pressure, 00:25:48.780 --> 00:25:52.590 align:middle line:84% so that you can get pressure variation, directly, I think. 00:25:52.590 --> 00:25:55.740 align:middle line:84% [INAUDIBLE] But it's low-resolution. 00:25:55.740 --> 00:25:57.480 align:middle line:84% So I haven't seen anything that's 00:25:57.480 --> 00:25:59.615 align:middle line:84% high-resolution like ours with fingerprints. 00:25:59.615 --> 00:26:00.990 align:middle line:84% RAMESH RASKAR: They can do a pan, 00:26:00.990 --> 00:26:06.835 align:middle line:84% but they cannot do [? reaches. ?] 00:26:06.835 --> 00:26:08.085 align:middle line:90% MICAH KIMO JOHNSON: All right. 00:26:08.085 --> 00:26:10.068 align:middle line:84% RAMESH RASKAR: Any other questions for Kimo? 00:26:10.068 --> 00:26:11.110 align:middle line:90% MICAH KIMO JOHNSON: Yeah. 00:26:11.110 --> 00:26:12.943 align:middle line:84% AUDIENCE: How do you come up with the ideas? 00:26:12.943 --> 00:26:14.573 align:middle line:90% 00:26:14.573 --> 00:26:16.240 align:middle line:84% RAMESH RASKAR: So what was the question? 00:26:16.240 --> 00:26:18.407 align:middle line:84% MICAH KIMO JOHNSON: How do we come up with the idea? 00:26:18.407 --> 00:26:25.560 align:middle line:84% So I think it's a couple of things. 00:26:25.560 --> 00:26:30.390 align:middle line:84% Ted has young children, and he was fascinated with the idea 00:26:30.390 --> 00:26:34.837 align:middle line:84% that touch is actually a very important sense to infants. 00:26:34.837 --> 00:26:36.420 align:middle line:84% They're touching things to their lips, 00:26:36.420 --> 00:26:38.970 align:middle line:84% [INAUDIBLE] lot of sensory organs that they're touching. 00:26:38.970 --> 00:26:43.260 align:middle line:84% And touch is relatively unstudied 00:26:43.260 --> 00:26:45.180 align:middle line:84% compared to vision and other things. 00:26:45.180 --> 00:26:47.490 align:middle line:84% And one of the reasons is that there isn't a good touch 00:26:47.490 --> 00:26:51.978 align:middle line:84% sensor, something that measures touch the way skin does. 00:26:51.978 --> 00:26:53.520 align:middle line:84% It deforms when something presses it. 00:26:53.520 --> 00:26:55.590 align:middle line:84% And so he thought, well, how can I 00:26:55.590 --> 00:26:59.352 align:middle line:84% build a touch sensor, something that deforms like skin. 00:26:59.352 --> 00:27:01.560 align:middle line:84% I think at the same time he was having foot problems, 00:27:01.560 --> 00:27:04.760 align:middle line:84% so he was just separately buying these gels to make insoles. 00:27:04.760 --> 00:27:07.260 align:middle line:84% And then these two things mixed in his head, and he thought, 00:27:07.260 --> 00:27:08.340 align:middle line:90% oh, I'll make the gel. 00:27:08.340 --> 00:27:13.543 align:middle line:90% 00:27:13.543 --> 00:27:15.210 align:middle line:84% RAMESH RASKAR: Did you pass this around? 00:27:15.210 --> 00:27:16.293 align:middle line:90% MICAH KIMO JOHNSON: I did. 00:27:16.293 --> 00:27:17.010 align:middle line:90% Yeah. 00:27:17.010 --> 00:27:20.640 align:middle line:84% RAMESH RASKAR: I was fortunate enough to have one for myself. 00:27:20.640 --> 00:27:22.530 align:middle line:84% But if anybody wants to do a project on this, 00:27:22.530 --> 00:27:23.613 align:middle line:90% you can collaborate with-- 00:27:23.613 --> 00:27:24.940 align:middle line:90% [INTERPOSING VOICES] 00:27:24.940 --> 00:27:25.500 align:middle line:84% MICAH KIMO JOHNSON: Or if you want 00:27:25.500 --> 00:27:27.300 align:middle line:84% to actually play with the demo in person, 00:27:27.300 --> 00:27:29.887 align:middle line:90% we have it set up in our lab. 00:27:29.887 --> 00:27:31.470 align:middle line:84% RAMESH RASKAR: Or he could be a mentor 00:27:31.470 --> 00:27:33.612 align:middle line:84% if you want it to be a final projects in the class. 00:27:33.612 --> 00:27:35.820 align:middle line:84% AUDIENCE: So [INAUDIBLE] have some idea in the space? 00:27:35.820 --> 00:27:36.090 align:middle line:90% It can-- 00:27:36.090 --> 00:27:36.660 align:middle line:84% MICAH KIMO JOHNSON: We are thinking 00:27:36.660 --> 00:27:38.550 align:middle line:84% of what the next generation of this will be, 00:27:38.550 --> 00:27:42.630 align:middle line:84% and we're shooting for a Siggraph project based on this. 00:27:42.630 --> 00:27:47.615 align:middle line:90% 00:27:47.615 --> 00:27:49.240 align:middle line:84% RAMESH RASKAR: All of our [INAUDIBLE],, 00:27:49.240 --> 00:27:52.000 align:middle line:84% that's why the classes in the fall. 00:27:52.000 --> 00:27:54.880 align:middle line:84% And the final projects are due 4th of December. 00:27:54.880 --> 00:27:56.470 align:middle line:84% We have one and half months to watch. 00:27:56.470 --> 00:27:59.620 align:middle line:90% 00:27:59.620 --> 00:28:00.250 align:middle line:90% Thanks, Kimo. 00:28:00.250 --> 00:28:01.800 align:middle line:90% MICAH KIMO JOHNSON: Yeah.