WEBVTT 00:00:00.000 --> 00:00:02.490 align:middle line:84% The following content is provided under a Creative 00:00:02.490 --> 00:00:03.730 align:middle line:90% Commons license. 00:00:03.730 --> 00:00:06.120 align:middle line:84% Your support will help MIT OpenCourseWare 00:00:06.120 --> 00:00:10.090 align:middle line:84% continue to offer high-quality educational resources for free. 00:00:10.090 --> 00:00:12.720 align:middle line:84% To make a donation or to view additional materials 00:00:12.720 --> 00:00:16.650 align:middle line:84% from hundreds of MIT courses, visit MIT OpenCourseWare 00:00:16.650 --> 00:00:20.300 align:middle line:90% at ocw.mit.edu. 00:00:20.300 --> 00:00:22.970 align:middle line:84% PROFESSOR: Yeah, and again, just go slow, because we just 00:00:22.970 --> 00:00:24.790 align:middle line:90% talked about this [INAUDIBLE]. 00:00:24.790 --> 00:00:27.590 align:middle line:84% MATT HIRSCH: So this is a project that-- 00:00:27.590 --> 00:00:30.950 align:middle line:84% this presentation actually came from SIGGRAPH this August. 00:00:30.950 --> 00:00:32.820 align:middle line:84% I'm going to skip around a little bit. 00:00:32.820 --> 00:00:34.310 align:middle line:84% PROFESSOR: You might want to say just a couple of sentences 00:00:34.310 --> 00:00:35.330 align:middle line:90% about-- 00:00:35.330 --> 00:00:38.450 align:middle line:84% Mike had a question about, how did class projects 00:00:38.450 --> 00:00:41.360 align:middle line:84% start in this class, and how did they evolve? 00:00:41.360 --> 00:00:44.060 align:middle line:84% And I said you'd be a good example of how it started. 00:00:44.060 --> 00:00:44.715 align:middle line:84% So just say a couple of sentences 00:00:44.715 --> 00:00:45.615 align:middle line:90% about how you started-- 00:00:45.615 --> 00:00:46.490 align:middle line:84% MATT HIRSCH: Shall I start with that? 00:00:46.490 --> 00:00:46.910 align:middle line:90% PROFESSOR: Yeah. 00:00:46.910 --> 00:00:47.577 align:middle line:90% MATT HIRSCH: OK. 00:00:47.577 --> 00:00:53.030 align:middle line:84% So I guess I had taken Ramesh's class in the spring term 00:00:53.030 --> 00:00:57.470 align:middle line:84% before your first class here and had been really interested 00:00:57.470 --> 00:00:59.520 align:middle line:90% in camera work. 00:00:59.520 --> 00:01:04.180 align:middle line:84% And so Ramesh and I had been thinking about projects 00:01:04.180 --> 00:01:05.110 align:middle line:90% after that class. 00:01:05.110 --> 00:01:07.330 align:middle line:84% And over the summer, I kind of tried 00:01:07.330 --> 00:01:12.240 align:middle line:84% to develop some ideas that might lead to a thesis. 00:01:12.240 --> 00:01:15.790 align:middle line:84% And so I guess coming into this class, when I took it, 00:01:15.790 --> 00:01:20.230 align:middle line:84% I had the intention to hopefully develop some project that I did 00:01:20.230 --> 00:01:22.760 align:middle line:90% in here into some thesis work. 00:01:22.760 --> 00:01:26.630 align:middle line:84% And I guess that is a good place to start, 00:01:26.630 --> 00:01:30.350 align:middle line:84% maybe, if you're interested in really developing 00:01:30.350 --> 00:01:32.480 align:middle line:84% a project beyond the scope of the class, 00:01:32.480 --> 00:01:35.390 align:middle line:84% is to have some other good motivation to do that, 00:01:35.390 --> 00:01:37.310 align:middle line:90% like a thesis. 00:01:37.310 --> 00:01:43.620 align:middle line:84% But yeah, then when I did the final project for this class, 00:01:43.620 --> 00:01:47.400 align:middle line:84% I started thinking about some of the work that Ramesh had 00:01:47.400 --> 00:01:51.330 align:middle line:84% presented and working with a post-do-- 00:01:51.330 --> 00:01:55.920 align:middle line:84% or a [INAUDIBLE] who's soon to be graduating, 00:01:55.920 --> 00:01:59.370 align:middle line:84% who has done some work with Ramesh back at [INAUDIBLE],, 00:01:59.370 --> 00:02:02.790 align:middle line:84% and really got excited about it and so 00:02:02.790 --> 00:02:05.400 align:middle line:84% ended up developing that into my thesis. 00:02:05.400 --> 00:02:07.510 align:middle line:84% And that's where this project came from. 00:02:07.510 --> 00:02:08.220 align:middle line:90% So-- 00:02:08.220 --> 00:02:09.150 align:middle line:90% PROFESSOR: It's a great example. 00:02:09.150 --> 00:02:10.608 align:middle line:84% As you'll see, it's a great example 00:02:10.608 --> 00:02:13.170 align:middle line:84% of beautiful theory, beautiful implementation, 00:02:13.170 --> 00:02:17.020 align:middle line:90% and impactful implementation. 00:02:17.020 --> 00:02:20.250 align:middle line:84% So those three [INAUDIBLE] that we have-- novelty, execution, 00:02:20.250 --> 00:02:23.057 align:middle line:90% and impact, are [INAUDIBLE]. 00:02:23.057 --> 00:02:25.140 align:middle line:84% It's surprising he didn't win the best [INAUDIBLE] 00:02:25.140 --> 00:02:30.781 align:middle line:84% award., even by a popular vote [INAUDIBLE].. 00:02:30.781 --> 00:02:32.787 align:middle line:90% That's the perfect award. 00:02:32.787 --> 00:02:34.620 align:middle line:84% MATT HIRSCH: Well, we had some really cool-- 00:02:34.620 --> 00:02:36.300 align:middle line:84% have you shown them the projects from last-- 00:02:36.300 --> 00:02:37.758 align:middle line:84% PROFESSOR: Yeah, just very briefly. 00:02:37.758 --> 00:02:41.070 align:middle line:84% But [INAUDIBLE],, we should talk over them more. 00:02:41.070 --> 00:02:42.840 align:middle line:84% MATT HIRSCH: Yeah, so I guess I'll just 00:02:42.840 --> 00:02:45.510 align:middle line:90% get started with this. 00:02:45.510 --> 00:02:49.200 align:middle line:84% The goal here is to think about new ways of interacting 00:02:49.200 --> 00:02:51.270 align:middle line:90% with a thin-screen device. 00:02:51.270 --> 00:02:56.880 align:middle line:84% And imagine if your screen could basically not only support 00:02:56.880 --> 00:03:02.610 align:middle line:84% on-screen touch interaction, but this type of off-screen hover 00:03:02.610 --> 00:03:05.760 align:middle line:84% gesture interaction that we think about here at the Media 00:03:05.760 --> 00:03:07.420 align:middle line:90% Lab in a lot of contexts. 00:03:07.420 --> 00:03:09.300 align:middle line:84% And so you see that brief example 00:03:09.300 --> 00:03:12.142 align:middle line:84% where I'm able to lift up my hand and interact in free space 00:03:12.142 --> 00:03:13.350 align:middle line:90% right in front of the screen. 00:03:13.350 --> 00:03:16.080 align:middle line:84% And here's another example where I'm manipulating an object, 00:03:16.080 --> 00:03:19.110 align:middle line:84% and I select it by touching the screen in traditional mode, 00:03:19.110 --> 00:03:22.230 align:middle line:84% but can pull my hand away and rotate it around 00:03:22.230 --> 00:03:23.430 align:middle line:90% like Luke Skywalker. 00:03:23.430 --> 00:03:26.310 align:middle line:84% PROFESSOR: Use the laser pointer. 00:03:26.310 --> 00:03:28.690 align:middle line:90% MATT HIRSCH: OK. 00:03:28.690 --> 00:03:35.050 align:middle line:84% So this is kind of inspired by three emerging areas 00:03:35.050 --> 00:03:37.640 align:middle line:90% in HCI and camera research. 00:03:37.640 --> 00:03:42.610 align:middle line:84% One of them is this new class of light-sensitive display. 00:03:42.610 --> 00:03:45.482 align:middle line:84% You can mention if you've covered some of this stuff, 00:03:45.482 --> 00:03:46.190 align:middle line:90% and I'll skip it. 00:03:46.190 --> 00:03:50.460 align:middle line:84% PROFESSOR: Not how it works, so this is perfect. 00:03:50.460 --> 00:03:52.820 align:middle line:84% MATT HIRSCH: You all know how LCDs function. 00:03:52.820 --> 00:03:56.420 align:middle line:84% They're basically a matrix of transistors. 00:03:56.420 --> 00:04:00.290 align:middle line:84% There's a couple of companies that are taking that transistor 00:04:00.290 --> 00:04:04.160 align:middle line:84% matrix and embedding a single extra transistor that's 00:04:04.160 --> 00:04:06.450 align:middle line:84% optically sensitive into the matrix 00:04:06.450 --> 00:04:10.610 align:middle line:84% so that the net result is the entire LCD is 00:04:10.610 --> 00:04:13.640 align:middle line:90% a large-area optical sensor. 00:04:13.640 --> 00:04:15.790 align:middle line:84% And they're using these for touch interactions. 00:04:15.790 --> 00:04:17.150 align:middle line:90% So that was one inspiration. 00:04:17.150 --> 00:04:20.930 align:middle line:84% And then the second, of course, is depth cameras, which you 00:04:20.930 --> 00:04:22.340 align:middle line:90% may have covered a little bit. 00:04:22.340 --> 00:04:24.132 align:middle line:84% There are a couple of different techniques, 00:04:24.132 --> 00:04:26.660 align:middle line:84% but the upshot is that these cameras not only produce 00:04:26.660 --> 00:04:29.900 align:middle line:84% an RGB image of a scene, but a map of the scene 00:04:29.900 --> 00:04:32.900 align:middle line:84% where, for each pixel, you have a measurement of the distance 00:04:32.900 --> 00:04:35.640 align:middle line:90% from the camera to some object. 00:04:35.640 --> 00:04:40.170 align:middle line:84% And the third, I guess, is this sort of ubiquitous multitouch 00:04:40.170 --> 00:04:42.660 align:middle line:84% display, which has been popularized by Jeff Han 00:04:42.660 --> 00:04:45.480 align:middle line:84% here and the CNN wall, and of course 00:04:45.480 --> 00:04:48.090 align:middle line:84% the iPhone and a lot of other consumer electronics devices. 00:04:48.090 --> 00:04:49.590 align:middle line:84% PROFESSOR: [INAUDIBLE] other people, 00:04:49.590 --> 00:04:53.160 align:middle line:90% that makes a good use right now. 00:04:53.160 --> 00:04:56.760 align:middle line:84% MATT HIRSCH: So we're kind of inspired by the ability 00:04:56.760 --> 00:04:59.730 align:middle line:84% to so easily interact or so intuitively interact 00:04:59.730 --> 00:05:02.550 align:middle line:84% with information on the screen and thinking about, 00:05:02.550 --> 00:05:05.290 align:middle line:84% can we take that one step further? 00:05:05.290 --> 00:05:09.120 align:middle line:84% So what if you could combine all of these things, 00:05:09.120 --> 00:05:15.180 align:middle line:84% basically, and be able to, because it's an LCD, 00:05:15.180 --> 00:05:16.890 align:middle line:84% build it all into a thin package, 00:05:16.890 --> 00:05:19.467 align:middle line:84% but also, because it's a depth camera, 00:05:19.467 --> 00:05:21.550 align:middle line:84% be able to track hands out in front of the screen? 00:05:21.550 --> 00:05:22.905 align:middle line:90% So that's where we started. 00:05:22.905 --> 00:05:27.000 align:middle line:90% 00:05:27.000 --> 00:05:28.730 align:middle line:84% The benefit of that, of course, would 00:05:28.730 --> 00:05:31.080 align:middle line:84% be that you can bring this depth sensibility 00:05:31.080 --> 00:05:34.320 align:middle line:84% to all sorts of consumer electronics type of devices 00:05:34.320 --> 00:05:36.660 align:middle line:84% that don't have it or it wouldn't even 00:05:36.660 --> 00:05:38.070 align:middle line:84% be possible to think about today, 00:05:38.070 --> 00:05:40.140 align:middle line:90% like an iPhone or a laptop. 00:05:40.140 --> 00:05:42.810 align:middle line:84% So to give you a brief overview of the results-- 00:05:42.810 --> 00:05:47.050 align:middle line:84% I think Ramesh may have covered some of this. 00:05:47.050 --> 00:05:51.455 align:middle line:84% This is basically a light field, and you've probably seen-- 00:05:51.455 --> 00:05:52.830 align:middle line:84% PROFESSOR: We just covered today. 00:05:52.830 --> 00:05:53.580 align:middle line:90% MATT HIRSCH: Yeah. 00:05:53.580 --> 00:05:54.910 align:middle line:90% PROFESSOR: [INAUDIBLE] 00:05:54.910 --> 00:05:57.840 align:middle line:84% MATT HIRSCH: So this is what we were able to capture. 00:05:57.840 --> 00:06:00.878 align:middle line:84% And there, you see a synthetically refocused image. 00:06:00.878 --> 00:06:02.420 align:middle line:84% You guys, I think you just did your-- 00:06:02.420 --> 00:06:03.120 align:middle line:84% PROFESSOR: That's our synthetic aperture. 00:06:03.120 --> 00:06:05.005 align:middle line:84% MATT HIRSCH: --synthetic aperture project. 00:06:05.005 --> 00:06:07.380 align:middle line:84% So this is one application of that, where we're basically 00:06:07.380 --> 00:06:09.960 align:middle line:84% taking the set of images captured by the light field, 00:06:09.960 --> 00:06:12.510 align:middle line:84% synthetically refocusing them, and extracting a depth. 00:06:12.510 --> 00:06:15.230 align:middle line:90% 00:06:15.230 --> 00:06:21.460 align:middle line:84% So just to give you some of the-- 00:06:21.460 --> 00:06:24.130 align:middle line:84% I guess to think about how to adapt 00:06:24.130 --> 00:06:26.520 align:middle line:84% one of these optical touchscreens, 00:06:26.520 --> 00:06:29.380 align:middle line:84% you know, it works a lot like a document scanner, 00:06:29.380 --> 00:06:33.160 align:middle line:84% where you have this array of pixels without lenses. 00:06:33.160 --> 00:06:37.780 align:middle line:84% An object that's touching that layer of pixels 00:06:37.780 --> 00:06:40.030 align:middle line:84% can have a sharp image made because you 00:06:40.030 --> 00:06:43.150 align:middle line:84% have a one-to-one correspondence between a point in the scene 00:06:43.150 --> 00:06:46.030 align:middle line:84% and a point on your sensor, just by virtue of their being 00:06:46.030 --> 00:06:47.110 align:middle line:90% so close together. 00:06:47.110 --> 00:06:49.480 align:middle line:84% But of course, when you pull your hand away 00:06:49.480 --> 00:06:51.820 align:middle line:84% from that sensor, you no longer have 00:06:51.820 --> 00:06:54.370 align:middle line:90% that one-to-one correspondence. 00:06:54.370 --> 00:06:57.190 align:middle line:84% Rays can travel from this object to many different pixels, 00:06:57.190 --> 00:07:01.100 align:middle line:90% and so you get a blur. 00:07:01.100 --> 00:07:03.040 align:middle line:84% So I guess our approach, then, is 00:07:03.040 --> 00:07:06.430 align:middle line:84% to think about a way we can basically 00:07:06.430 --> 00:07:08.290 align:middle line:84% bring that one-to-one correspondence back 00:07:08.290 --> 00:07:11.420 align:middle line:84% without using any kind of lenses. 00:07:11.420 --> 00:07:15.160 align:middle line:84% And so what we do is separate the sensor by a small margin 00:07:15.160 --> 00:07:17.650 align:middle line:84% from the display and then display 00:07:17.650 --> 00:07:20.020 align:middle line:84% one of these types of masks that Ramesh was just 00:07:20.020 --> 00:07:21.410 align:middle line:90% describing to you guys. 00:07:21.410 --> 00:07:23.590 align:middle line:84% So in this case, what we're considering 00:07:23.590 --> 00:07:27.250 align:middle line:84% is using the LCD as both a display device 00:07:27.250 --> 00:07:30.790 align:middle line:84% for the user to see the images, like a typical LCD screen, 00:07:30.790 --> 00:07:35.200 align:middle line:84% and also as a device to create one of these masks 00:07:35.200 --> 00:07:36.970 align:middle line:84% that Ramesh was describing in order 00:07:36.970 --> 00:07:41.390 align:middle line:84% to encode the scene in a way that we can decode in software. 00:07:41.390 --> 00:07:45.170 align:middle line:84% So here's an idea of what that device might look like. 00:07:45.170 --> 00:07:49.960 align:middle line:84% You have your LCD screen here, and some distance behind it, 00:07:49.960 --> 00:07:51.040 align:middle line:90% you have a sensor layer. 00:07:51.040 --> 00:07:52.760 align:middle line:90% And out here, you have objects. 00:07:52.760 --> 00:07:55.210 align:middle line:90% And you can actually decode-- 00:07:55.210 --> 00:07:57.070 align:middle line:84% the vision is that in this thin device, 00:07:57.070 --> 00:08:01.258 align:middle line:84% you could then decode this object, process the imagery, 00:08:01.258 --> 00:08:02.800 align:middle line:84% and then re-display it on the screen, 00:08:02.800 --> 00:08:07.810 align:middle line:84% maybe in a modified way, or in the case that I'm describing, 00:08:07.810 --> 00:08:11.280 align:middle line:84% to interact with the screen gesturally. 00:08:11.280 --> 00:08:14.100 align:middle line:84% And so this is a kind of interesting device. 00:08:14.100 --> 00:08:16.380 align:middle line:84% And I like to think about the pinhole analogy, 00:08:16.380 --> 00:08:17.700 align:middle line:90% because it's very intuitive. 00:08:17.700 --> 00:08:20.310 align:middle line:90% 00:08:20.310 --> 00:08:23.130 align:middle line:84% Just looking at that mask, it's not quite clear to me 00:08:23.130 --> 00:08:26.130 align:middle line:84% what you get just from an intuitive sense. 00:08:26.130 --> 00:08:30.030 align:middle line:84% But the pinhole makes a very easy case to think about. 00:08:30.030 --> 00:08:32.250 align:middle line:84% If you imagine tiling those pinholes 00:08:32.250 --> 00:08:35.309 align:middle line:84% all across the aperture or all across the LCD, 00:08:35.309 --> 00:08:40.107 align:middle line:84% you get, basically, many tiny cameras covering the screen. 00:08:40.107 --> 00:08:42.690 align:middle line:84% And each of those cameras has a slightly different perspective 00:08:42.690 --> 00:08:43.299 align:middle line:90% of the scene. 00:08:43.299 --> 00:08:45.660 align:middle line:84% And putting those together, that's 00:08:45.660 --> 00:08:49.120 align:middle line:84% basically a light field that you're capturing right there. 00:08:49.120 --> 00:08:51.360 align:middle line:84% And the interesting thing in thinking about this 00:08:51.360 --> 00:08:54.390 align:middle line:90% is when we-- 00:08:54.390 --> 00:08:57.050 align:middle line:84% this is not a normal type of camera 00:08:57.050 --> 00:09:01.290 align:middle line:84% that you're used to using to capture pictures of birthday 00:09:01.290 --> 00:09:02.310 align:middle line:90% parties or whatever. 00:09:02.310 --> 00:09:06.540 align:middle line:84% It's going to produce a pretty strange-looking image 00:09:06.540 --> 00:09:10.880 align:middle line:84% because, if you think about how you would image an object out 00:09:10.880 --> 00:09:14.740 align:middle line:84% in front of the device, first of all, 00:09:14.740 --> 00:09:17.520 align:middle line:84% it produces an orthographic image. 00:09:17.520 --> 00:09:19.770 align:middle line:90% There's no perspective here. 00:09:19.770 --> 00:09:21.930 align:middle line:84% Let's say this is one of my pinholes, right? 00:09:21.930 --> 00:09:25.170 align:middle line:84% If I want to image something that's 00:09:25.170 --> 00:09:27.030 align:middle line:84% off to the left of my device, what I do 00:09:27.030 --> 00:09:30.090 align:middle line:84% is I take maybe the pixel on my sensor 00:09:30.090 --> 00:09:34.200 align:middle line:84% that's over here for each of these tiled pinhole cameras. 00:09:34.200 --> 00:09:37.380 align:middle line:84% And that ray basically goes out into the scene 00:09:37.380 --> 00:09:40.930 align:middle line:84% and is projected out in a straight line. 00:09:40.930 --> 00:09:43.110 align:middle line:90% So there's no perspective. 00:09:43.110 --> 00:09:46.023 align:middle line:90% It's all parallel rays. 00:09:46.023 --> 00:09:47.940 align:middle line:84% And the other interesting thing to think about 00:09:47.940 --> 00:09:50.310 align:middle line:84% is, you can see that the resolution actually 00:09:50.310 --> 00:09:50.940 align:middle line:90% does decrease. 00:09:50.940 --> 00:09:54.360 align:middle line:84% Imagine this blue is the size of my pixel. 00:09:54.360 --> 00:09:56.370 align:middle line:84% I can project that pixel out into the scene 00:09:56.370 --> 00:09:59.340 align:middle line:84% to see what I'm measuring, and you 00:09:59.340 --> 00:10:01.650 align:middle line:84% can see that as an object gets further away, 00:10:01.650 --> 00:10:06.670 align:middle line:84% my pixel, relative to the size of the object, is increasing. 00:10:06.670 --> 00:10:09.300 align:middle line:90% So just a brief tangent there. 00:10:09.300 --> 00:10:12.270 align:middle line:84% But I'll just show you a couple more ideas 00:10:12.270 --> 00:10:14.790 align:middle line:90% of how this might go. 00:10:14.790 --> 00:10:18.720 align:middle line:84% You might think about being able to navigate spaces by just 00:10:18.720 --> 00:10:22.200 align:middle line:90% moving your hand in free space. 00:10:22.200 --> 00:10:25.590 align:middle line:90% And because it's a-- 00:10:25.590 --> 00:10:28.890 align:middle line:84% think-- because it's optically sensitive, 00:10:28.890 --> 00:10:30.260 align:middle line:90% you can think about-- 00:10:30.260 --> 00:10:31.446 align:middle line:90% where is it? 00:10:31.446 --> 00:10:37.070 align:middle line:90% 00:10:37.070 --> 00:10:40.520 align:middle line:84% You can think about doing a demo like this, where I'm actually 00:10:40.520 --> 00:10:43.880 align:middle line:84% taking a real-world flashlight and projecting it 00:10:43.880 --> 00:10:45.890 align:middle line:90% into a virtual scene. 00:10:45.890 --> 00:10:52.220 align:middle line:84% So I'm taking real light and mapping the light field that I 00:10:52.220 --> 00:10:54.620 align:middle line:90% capture into a virtual world. 00:10:54.620 --> 00:10:57.170 align:middle line:84% This is an interesting mixed reality. 00:10:57.170 --> 00:10:59.480 align:middle line:90% PROFESSOR: Is it the end? 00:10:59.480 --> 00:11:01.160 align:middle line:84% MATT HIRSCH: Yeah, it seems a little-- 00:11:01.160 --> 00:11:04.160 align:middle line:84% you can see there's a hand here holding a flashlight. 00:11:04.160 --> 00:11:08.820 align:middle line:84% And that's actually shining light into this virtual world. 00:11:08.820 --> 00:11:16.103 align:middle line:84% So I guess maybe I'm going to skip over this a little bit. 00:11:16.103 --> 00:11:17.520 align:middle line:84% But you can think-- there are lots 00:11:17.520 --> 00:11:22.080 align:middle line:84% of ways to accomplish this, or accomplish parts of what I'm 00:11:22.080 --> 00:11:25.620 align:middle line:84% describing, that don't involve using the method 00:11:25.620 --> 00:11:26.460 align:middle line:90% that I'm describing. 00:11:26.460 --> 00:11:30.870 align:middle line:84% But if you look at the entire package-- 00:11:30.870 --> 00:11:32.670 align:middle line:84% putting it in a thin device, being 00:11:32.670 --> 00:11:37.170 align:middle line:84% able to capture both touch and gesture-- 00:11:37.170 --> 00:11:41.050 align:middle line:84% I think it becomes a pretty compelling idea. 00:11:41.050 --> 00:11:47.650 align:middle line:84% So I think Ramesh has described a little bit about light fields 00:11:47.650 --> 00:11:48.150 align:middle line:90% before. 00:11:48.150 --> 00:11:51.900 align:middle line:84% I'm just going to cover the basic ingredients 00:11:51.900 --> 00:11:54.120 align:middle line:84% that I've found very helpful to understand 00:11:54.120 --> 00:11:55.620 align:middle line:90% the theory behind this. 00:11:55.620 --> 00:11:58.853 align:middle line:84% And if I'm covering something that you've already seen, 00:11:58.853 --> 00:11:59.520 align:middle line:90% you can stop me. 00:11:59.520 --> 00:12:04.110 align:middle line:84% But in a light field-- let's just imagine the 2D case-- when 00:12:04.110 --> 00:12:06.120 align:middle line:84% you have a ray, the basic idea is 00:12:06.120 --> 00:12:07.710 align:middle line:84% you want to parameterize this ray. 00:12:07.710 --> 00:12:11.760 align:middle line:84% You want to describe a set of rays in a new space. 00:12:11.760 --> 00:12:14.850 align:middle line:84% So I have a ray that has some intersection with a sensor 00:12:14.850 --> 00:12:16.020 align:middle line:90% plane. 00:12:16.020 --> 00:12:17.710 align:middle line:90% And it intersects with an angle. 00:12:17.710 --> 00:12:20.280 align:middle line:84% So if I just plot the point where it intersects 00:12:20.280 --> 00:12:23.500 align:middle line:84% and the angle at which it intersects in this new space, 00:12:23.500 --> 00:12:24.960 align:middle line:90% this is a light field. 00:12:24.960 --> 00:12:29.430 align:middle line:84% And you can see, if I have a whole set of rays, 00:12:29.430 --> 00:12:31.620 align:middle line:84% that creates a sort of line in the light field 00:12:31.620 --> 00:12:32.910 align:middle line:90% space over there. 00:12:32.910 --> 00:12:33.948 align:middle line:90% And then if I have-- 00:12:33.948 --> 00:12:35.490 align:middle line:84% oh, well, this is actually important. 00:12:35.490 --> 00:12:39.110 align:middle line:90% If I-- oh, that's weird. 00:12:39.110 --> 00:12:41.860 align:middle line:90% 00:12:41.860 --> 00:12:46.591 align:middle line:84% I imported this from PowerPoint, so not all of these-- 00:12:46.591 --> 00:12:50.680 align:middle line:84% OK, well, it's important to note that one of these 00:12:50.680 --> 00:12:52.330 align:middle line:90% is what a real sensor measures. 00:12:52.330 --> 00:12:54.430 align:middle line:84% If you take a real sensor in the real world 00:12:54.430 --> 00:12:57.190 align:middle line:84% and just expose it to light without any lens 00:12:57.190 --> 00:12:59.500 align:middle line:84% or anything in front of it, it integrates rays 00:12:59.500 --> 00:13:00.980 align:middle line:90% from all directions. 00:13:00.980 --> 00:13:03.850 align:middle line:84% And so what you're actually measuring in the light field 00:13:03.850 --> 00:13:07.386 align:middle line:90% space is one of these lines. 00:13:07.386 --> 00:13:11.450 align:middle line:84% So this is what a sensor array might look like. 00:13:11.450 --> 00:13:16.490 align:middle line:84% Now, if I think about the frequency domain 00:13:16.490 --> 00:13:21.350 align:middle line:84% picture of this space, I'll have some light field 00:13:21.350 --> 00:13:24.260 align:middle line:84% here that I'm taking projections through, basically. 00:13:24.260 --> 00:13:26.150 align:middle line:84% And that light field has some spectrum 00:13:26.150 --> 00:13:28.190 align:middle line:90% in the frequency domain. 00:13:28.190 --> 00:13:31.160 align:middle line:84% And as I mentioned, these lines are projections 00:13:31.160 --> 00:13:32.160 align:middle line:90% through the light field. 00:13:32.160 --> 00:13:36.290 align:middle line:84% So there's something called the Fourier slice theorem, which 00:13:36.290 --> 00:13:38.960 align:middle line:84% basically says if I take a projection through a function 00:13:38.960 --> 00:13:41.210 align:middle line:84% in this domain, I'm actually taking 00:13:41.210 --> 00:13:47.640 align:middle line:84% a slice through my spectrum in the frequency domain. 00:13:47.640 --> 00:13:49.460 align:middle line:84% So I guess the important thought trap here 00:13:49.460 --> 00:13:52.370 align:middle line:84% is that if I am actually measuring 00:13:52.370 --> 00:13:55.730 align:middle line:84% this with my real-world sensor, in the frequency domain, 00:13:55.730 --> 00:13:57.910 align:middle line:84% I'm actually measuring only what's along this axis. 00:13:57.910 --> 00:13:59.660 align:middle line:84% So this is the only thing accessible to me 00:13:59.660 --> 00:14:03.140 align:middle line:90% with a real-world sensor. 00:14:03.140 --> 00:14:06.170 align:middle line:84% And of course, what I actually have over 00:14:06.170 --> 00:14:07.470 align:middle line:90% there is that whole spectrum. 00:14:07.470 --> 00:14:12.390 align:middle line:84% And so the question is, how can I access that data? 00:14:12.390 --> 00:14:15.440 align:middle line:84% So the next important thing to keep in mind when 00:14:15.440 --> 00:14:18.590 align:middle line:84% trying to understand this is the skew property 00:14:18.590 --> 00:14:21.410 align:middle line:84% of the light field, which basically 00:14:21.410 --> 00:14:25.040 align:middle line:84% says, if I'm going to plot a light field, 00:14:25.040 --> 00:14:27.140 align:middle line:84% I can either look at it from this perspective 00:14:27.140 --> 00:14:28.440 align:middle line:90% or from this perspective. 00:14:28.440 --> 00:14:31.750 align:middle line:84% So I can say, as my ray travels through free space, 00:14:31.750 --> 00:14:35.720 align:middle line:84% I'm going to just plot the light field that it creates over 00:14:35.720 --> 00:14:37.200 align:middle line:90% on the right side there. 00:14:37.200 --> 00:14:39.830 align:middle line:84% And you can see a kind of interesting effect 00:14:39.830 --> 00:14:46.010 align:middle line:84% where, as I add rays, a straight line from one position in space 00:14:46.010 --> 00:14:49.590 align:middle line:84% becomes a skewed line in another position. 00:14:49.590 --> 00:14:54.260 align:middle line:84% And then, as you may recall from physics or signal processing 00:14:54.260 --> 00:14:57.410 align:middle line:84% theory, if I perform a convolution 00:14:57.410 --> 00:15:03.150 align:middle line:84% function between any array of delta functions 00:15:03.150 --> 00:15:05.720 align:middle line:84% and some other arbitrary function, 00:15:05.720 --> 00:15:10.770 align:middle line:84% I get a tiled version of my arbitrary function. 00:15:10.770 --> 00:15:13.160 align:middle line:84% So I'll tie this all together, I promise. 00:15:13.160 --> 00:15:15.660 align:middle line:84% Just try to keep these things in mind as we go. 00:15:15.660 --> 00:15:20.047 align:middle line:90% And then I guess-- 00:15:20.047 --> 00:15:22.630 align:middle line:84% I don't know if Ramesh has used the term spatial heterodyning, 00:15:22.630 --> 00:15:25.360 align:middle line:84% but I think it's a kind of cool historical note 00:15:25.360 --> 00:15:30.490 align:middle line:84% that heterodyning is a word that comes from old radio 00:15:30.490 --> 00:15:31.330 align:middle line:90% broadcasts. 00:15:31.330 --> 00:15:32.200 align:middle line:90% And when we say-- 00:15:32.200 --> 00:15:35.200 align:middle line:90% 00:15:35.200 --> 00:15:37.900 align:middle line:84% it was really a technique that multiplied a voice 00:15:37.900 --> 00:15:41.350 align:middle line:84% signal by a high-frequency signal in order to transmit it. 00:15:41.350 --> 00:15:43.300 align:middle line:84% It'd basically shift that voice up 00:15:43.300 --> 00:15:45.940 align:middle line:84% into a radio spectrum that could be broadcast. 00:15:45.940 --> 00:15:48.280 align:middle line:84% And this is really what we're doing when 00:15:48.280 --> 00:15:50.110 align:middle line:90% we send a ray through a mass. 00:15:50.110 --> 00:15:56.648 align:middle line:84% We're actually multiplying that ray by some frequency pattern. 00:15:56.648 --> 00:15:58.690 align:middle line:84% Spatial frequency, in this case, instead of time, 00:15:58.690 --> 00:16:01.000 align:middle line:90% but similar principle. 00:16:01.000 --> 00:16:04.540 align:middle line:84% And so, to bring this all together now, 00:16:04.540 --> 00:16:08.350 align:middle line:84% imagine I create a mask that has a transform that 00:16:08.350 --> 00:16:09.080 align:middle line:90% looks like this. 00:16:09.080 --> 00:16:10.850 align:middle line:84% It's a series of delta functions. 00:16:10.850 --> 00:16:15.108 align:middle line:84% And because I'm multiplying it in the primal domain, 00:16:15.108 --> 00:16:16.900 align:middle line:84% in the frequency domain, I'm convolving it. 00:16:16.900 --> 00:16:19.120 align:middle line:84% So I'm convolving my mask spectrum 00:16:19.120 --> 00:16:21.860 align:middle line:84% with the light field spectrum that I want to measure. 00:16:21.860 --> 00:16:25.120 align:middle line:84% And so what you get is a tiled version 00:16:25.120 --> 00:16:26.890 align:middle line:90% of that light field spectrum. 00:16:26.890 --> 00:16:29.230 align:middle line:84% And remember that shift property. 00:16:29.230 --> 00:16:33.080 align:middle line:84% I've offset my mask from my sensor a little bit. 00:16:33.080 --> 00:16:37.960 align:middle line:84% So the mask-- well, the light field that I've created 00:16:37.960 --> 00:16:40.660 align:middle line:84% is actually going to be skewed by the time it 00:16:40.660 --> 00:16:42.010 align:middle line:90% reaches my sensor. 00:16:42.010 --> 00:16:45.850 align:middle line:84% And so now you see the really cool, insightful part 00:16:45.850 --> 00:16:49.390 align:middle line:84% of all of this is that if you look at what's on the fx axis 00:16:49.390 --> 00:16:53.710 align:middle line:84% down there, you can see, along this axis, 00:16:53.710 --> 00:16:55.630 align:middle line:84% I get different portions of the spectrum. 00:16:55.630 --> 00:17:00.040 align:middle line:84% So I've now created a way to measure 00:17:00.040 --> 00:17:04.780 align:middle line:84% pieces of that spectrum on my real-world sensor. 00:17:04.780 --> 00:17:06.700 align:middle line:84% So I can just rearrange those things 00:17:06.700 --> 00:17:11.736 align:middle line:84% and reconstruct a good portion of my light fields back to me. 00:17:11.736 --> 00:17:14.319 align:middle line:84% So the important thing to keep in mind, or one important thing 00:17:14.319 --> 00:17:15.910 align:middle line:84% to keep in mind, which I guess goes 00:17:15.910 --> 00:17:18.035 align:middle line:84% towards some of the discussion you guys were having 00:17:18.035 --> 00:17:21.280 align:middle line:84% before I began speaking, is that you really have to make sure 00:17:21.280 --> 00:17:23.349 align:middle line:84% these light field spectral copies are 00:17:23.349 --> 00:17:26.410 align:middle line:84% band limited so that they won't interfere with each other. 00:17:26.410 --> 00:17:31.580 align:middle line:84% And I guess that speaks to Ramesh's point from earlier. 00:17:31.580 --> 00:17:36.940 align:middle line:84% So in terms of building this actual prototype, 00:17:36.940 --> 00:17:39.230 align:middle line:84% it's all fine to do this in theory, 00:17:39.230 --> 00:17:42.190 align:middle line:84% but we want to actually do it in practice 00:17:42.190 --> 00:17:45.650 align:middle line:84% and see if what we're doing on paper really makes sense. 00:17:45.650 --> 00:17:48.190 align:middle line:84% So what we really want is, remember, 00:17:48.190 --> 00:17:52.720 align:middle line:84% an LCD separated from a large-area sensor 00:17:52.720 --> 00:17:53.770 align:middle line:90% by a small margin. 00:17:53.770 --> 00:17:55.963 align:middle line:84% But that's a really difficult thing to get. 00:17:55.963 --> 00:17:57.880 align:middle line:84% These things will be out there in the consumer 00:17:57.880 --> 00:18:00.380 align:middle line:84% market in the near future, but they're not right now. 00:18:00.380 --> 00:18:02.860 align:middle line:84% So what we actually ended up doing 00:18:02.860 --> 00:18:08.050 align:middle line:84% was taking a couple of cameras and simply imaging a diffuser. 00:18:08.050 --> 00:18:10.420 align:middle line:84% Much like the movie screen shows you 00:18:10.420 --> 00:18:13.990 align:middle line:84% a slice of the light passing through a certain space, 00:18:13.990 --> 00:18:17.980 align:middle line:84% a diffuser will just show us optically 00:18:17.980 --> 00:18:21.890 align:middle line:84% what we would like to measure electronically in this plane. 00:18:21.890 --> 00:18:25.870 align:middle line:84% So here are the actual cameras that we 00:18:25.870 --> 00:18:31.000 align:middle line:90% used, and here's one prototype. 00:18:31.000 --> 00:18:33.010 align:middle line:84% I have, actually, a slightly newer one now, 00:18:33.010 --> 00:18:36.275 align:middle line:84% but the LCD was sitting in this screen, and this is a diffuser. 00:18:36.275 --> 00:18:39.320 align:middle line:90% 00:18:39.320 --> 00:18:42.847 align:middle line:84% And then I'll just quickly run through the software pipeline 00:18:42.847 --> 00:18:43.430 align:middle line:90% that we wrote. 00:18:43.430 --> 00:18:45.740 align:middle line:84% I won't spend too much time dwelling on this, 00:18:45.740 --> 00:18:49.340 align:middle line:84% but the basic idea is we want a time multiplex 00:18:49.340 --> 00:18:52.880 align:middle line:84% between displaying an image for the user to see 00:18:52.880 --> 00:18:55.250 align:middle line:84% and illuminating that image from behind, 00:18:55.250 --> 00:18:58.250 align:middle line:84% and then switching to the mask mode 00:18:58.250 --> 00:19:00.237 align:middle line:84% where we don't want to illuminate it 00:19:00.237 --> 00:19:02.570 align:middle line:84% because we don't want to interfere with our measurement. 00:19:02.570 --> 00:19:07.250 align:middle line:84% And we want to actually capture the data that is 00:19:07.250 --> 00:19:09.690 align:middle line:90% being modulated by that mask. 00:19:09.690 --> 00:19:14.690 align:middle line:84% And so I'll just play a couple of videos of the data 00:19:14.690 --> 00:19:17.730 align:middle line:84% from different portions of that pipeline. 00:19:17.730 --> 00:19:20.450 align:middle line:84% So here, you see the actual [INAUDIBLE] 00:19:20.450 --> 00:19:22.210 align:middle line:84% code that we display on the screen. 00:19:22.210 --> 00:19:24.835 align:middle line:84% PROFESSOR: [INAUDIBLE] this one doesn't look like cosine masks. 00:19:24.835 --> 00:19:25.900 align:middle line:90% It's a real binary mask. 00:19:25.900 --> 00:19:27.650 align:middle line:84% MATT HIRSCH: Yeah, this is the binary mask 00:19:27.650 --> 00:19:28.970 align:middle line:90% that Ramesh was mentioning. 00:19:28.970 --> 00:19:30.890 align:middle line:90% And it actually turns out-- 00:19:30.890 --> 00:19:34.162 align:middle line:84% if you recall from my theory description, 00:19:34.162 --> 00:19:35.870 align:middle line:84% the only thing I mentioned about the mask 00:19:35.870 --> 00:19:40.760 align:middle line:84% was that it has a transform that's a series of deltas. 00:19:40.760 --> 00:19:44.210 align:middle line:84% So it turns out, if you tile any code, 00:19:44.210 --> 00:19:47.290 align:middle line:84% you can get a transform that ends up being 00:19:47.290 --> 00:19:49.457 align:middle line:90% a series of delta functions. 00:19:49.457 --> 00:19:51.290 align:middle line:84% Those functions will have different weights, 00:19:51.290 --> 00:19:55.700 align:middle line:84% but it's kind of the Fourier series effect, right? 00:19:55.700 --> 00:19:56.420 align:middle line:90% And-- 00:19:56.420 --> 00:19:57.980 align:middle line:84% PROFESSOR: If you tile anything, you'll get [? deltas. ?] 00:19:57.980 --> 00:19:59.030 align:middle line:90% MATT HIRSCH: Yeah. 00:19:59.030 --> 00:20:02.510 align:middle line:84% And I guess the reason that this mask was chosen 00:20:02.510 --> 00:20:06.410 align:middle line:84% was that it actually is sort of optimal in terms 00:20:06.410 --> 00:20:07.460 align:middle line:90% of light efficiency. 00:20:07.460 --> 00:20:10.350 align:middle line:84% So this mask allows 50% of the light to pass through, 00:20:10.350 --> 00:20:13.430 align:middle line:84% which is pretty remarkable considering 00:20:13.430 --> 00:20:17.750 align:middle line:84% we're reconstructing an image without a lens. 00:20:17.750 --> 00:20:21.453 align:middle line:84% Just for comparison, the pinhole allows something like 1% to 2%, 00:20:21.453 --> 00:20:23.370 align:middle line:84% depending on the size of the pinhole you used. 00:20:23.370 --> 00:20:25.950 align:middle line:84% And I think the cosine mask had 18%? 00:20:25.950 --> 00:20:26.785 align:middle line:90% PROFESSOR: 18%. 00:20:26.785 --> 00:20:28.910 align:middle line:84% And there was a question out there about difference 00:20:28.910 --> 00:20:30.400 align:middle line:90% with a pinhole and-- 00:20:30.400 --> 00:20:31.900 align:middle line:90% in this sense. 00:20:31.900 --> 00:20:34.640 align:middle line:90% That's between 1% and 50%. 00:20:34.640 --> 00:20:35.390 align:middle line:90% MATT HIRSCH: Yeah. 00:20:35.390 --> 00:20:40.190 align:middle line:84% So the data that our sensor captures, 00:20:40.190 --> 00:20:41.940 align:middle line:84% and if you were sitting behind the screen, 00:20:41.940 --> 00:20:43.523 align:middle line:84% this is what you would see, basically, 00:20:43.523 --> 00:20:46.670 align:middle line:84% if a hand is moving around here and touching the screen, 00:20:46.670 --> 00:20:47.630 align:middle line:90% hovering over it. 00:20:47.630 --> 00:20:51.350 align:middle line:84% And you can see the high-frequency noise, 00:20:51.350 --> 00:20:53.390 align:middle line:84% or it looks like noise, that the mask creates. 00:20:53.390 --> 00:20:55.698 align:middle line:84% And from that, we can decode this light field. 00:20:55.698 --> 00:20:57.740 align:middle line:84% PROFESSOR: [INAUDIBLE] were 20 pixels, you think? 00:20:57.740 --> 00:20:59.330 align:middle line:90% [INAUDIBLE] 00:20:59.330 --> 00:21:03.050 align:middle line:84% MATT HIRSCH: Yeah, so it's a 20 by 20 light field angularly, 00:21:03.050 --> 00:21:08.430 align:middle line:84% and then each little tile there is about 100 by 80 pixels. 00:21:08.430 --> 00:21:11.060 align:middle line:84% So you can see many views of that hand moving around, 00:21:11.060 --> 00:21:11.660 align:middle line:90% basically. 00:21:11.660 --> 00:21:15.230 align:middle line:84% And then from that light field, as I mentioned, for each frame, 00:21:15.230 --> 00:21:18.710 align:middle line:84% we get this stack of images, or we can refocus that light field 00:21:18.710 --> 00:21:19.978 align:middle line:90% at a number of depth. 00:21:19.978 --> 00:21:21.770 align:middle line:84% PROFESSOR: It's doing the refocusing, which 00:21:21.770 --> 00:21:23.840 align:middle line:84% is your first part of assignment, 00:21:23.840 --> 00:21:28.070 align:middle line:84% in real time from this 400 images, like 20 by 20? 00:21:28.070 --> 00:21:28.820 align:middle line:90% MATT HIRSCH: Yeah. 00:21:28.820 --> 00:21:29.240 align:middle line:90% PROFESSOR: Yeah. 00:21:29.240 --> 00:21:29.990 align:middle line:90% MATT HIRSCH: Yeah. 00:21:29.990 --> 00:21:34.280 align:middle line:84% And then once we have a refocused image, for each-- 00:21:34.280 --> 00:21:36.200 align:middle line:84% basically, we have a whole stack of images. 00:21:36.200 --> 00:21:38.240 align:middle line:84% We traverse that image and use a method 00:21:38.240 --> 00:21:40.130 align:middle line:84% called depth from focus where we basically 00:21:40.130 --> 00:21:43.303 align:middle line:84% look at the contrast in each of those images at each pixel. 00:21:43.303 --> 00:21:44.720 align:middle line:84% And from that, we get a depth map. 00:21:44.720 --> 00:21:48.065 align:middle line:84% And that's the basic ingredient into all of those interactions 00:21:48.065 --> 00:21:48.680 align:middle line:90% down there. 00:21:48.680 --> 00:21:50.763 align:middle line:84% PROFESSOR: So that's for extract credit, the depth 00:21:50.763 --> 00:21:52.580 align:middle line:90% from focusing your [? SIM. ?] 00:21:52.580 --> 00:21:55.790 align:middle line:84% MATT HIRSCH: So I had some videos in there, 00:21:55.790 --> 00:21:57.840 align:middle line:90% but they don't work in there. 00:21:57.840 --> 00:21:59.030 align:middle line:90% PROFESSOR: Any questions? 00:21:59.030 --> 00:22:00.380 align:middle line:84% MATT HIRSCH: So yeah, I think that's probably 00:22:00.380 --> 00:22:01.430 align:middle line:90% good [INAUDIBLE], right? 00:22:01.430 --> 00:22:02.242 align:middle line:84% PROFESSOR: Yeah, that's what I thought. 00:22:02.242 --> 00:22:02.360 align:middle line:90% Yeah. 00:22:02.360 --> 00:22:03.230 align:middle line:90% Yes? 00:22:03.230 --> 00:22:04.990 align:middle line:84% AUDIENCE: Are you doing this in real time? 00:22:04.990 --> 00:22:10.900 align:middle line:84% Because [INAUDIBLE] complication [INAUDIBLE].. 00:22:10.900 --> 00:22:14.605 align:middle line:84% MATT HIRSCH: Well, no, luckily, computers are very fast. 00:22:14.605 --> 00:22:17.290 align:middle line:90% 00:22:17.290 --> 00:22:21.260 align:middle line:84% And actually, there's a couple of even free Fourier transform 00:22:21.260 --> 00:22:21.760 align:middle line:90% libraries. 00:22:21.760 --> 00:22:23.110 align:middle line:90% One is called FFTW. 00:22:23.110 --> 00:22:27.430 align:middle line:84% And it optimizes itself to your hardware, 00:22:27.430 --> 00:22:29.948 align:middle line:90% and it can run very fast. 00:22:29.948 --> 00:22:32.340 align:middle line:84% AUDIENCE: Was this made by-- was it in MATLAB, or-- 00:22:32.340 --> 00:22:34.788 align:middle line:84% MATT HIRSCH: No, no, this was written in C. But it-- 00:22:34.788 --> 00:22:36.330 align:middle line:84% PROFESSOR: It's running in real time. 00:22:36.330 --> 00:22:38.190 align:middle line:84% You can interact with it in real time. 00:22:38.190 --> 00:22:39.750 align:middle line:90% MATT HIRSCH: Close to real time. 00:22:39.750 --> 00:22:41.125 align:middle line:84% AUDIENCE: Six frames in a second. 00:22:41.125 --> 00:22:43.470 align:middle line:90% [INAUDIBLE] 00:22:43.470 --> 00:22:46.322 align:middle line:84% MATT HIRSCH: So the demo runs at about 20 frames per second. 00:22:46.322 --> 00:22:48.030 align:middle line:84% PROFESSOR: It's not 60 frames per second. 00:22:48.030 --> 00:22:50.680 align:middle line:84% MATT HIRSCH: Yeah, we're hoping to improve that. 00:22:50.680 --> 00:22:52.770 align:middle line:84% AUDIENCE: How about latency from hand movement 00:22:52.770 --> 00:22:54.720 align:middle line:90% to actual movement? 00:22:54.720 --> 00:22:56.617 align:middle line:84% MATT HIRSCH: It's one frame or so. 00:22:56.617 --> 00:22:58.200 align:middle line:84% AUDIENCE: We're on a different system. 00:22:58.200 --> 00:23:01.210 align:middle line:90% 00:23:01.210 --> 00:23:05.380 align:middle line:84% MATT HIRSCH: Yeah, computers are very fast. 00:23:05.380 --> 00:23:09.280 align:middle line:84% The key is really to pick a Fourier transform 00:23:09.280 --> 00:23:12.430 align:middle line:84% that can be broken down into small prime factors, 00:23:12.430 --> 00:23:14.960 align:middle line:84% because you can implement that very quickly. 00:23:14.960 --> 00:23:18.690 align:middle line:84% But as long as you do that, it'll run real fast. 00:23:18.690 --> 00:23:19.620 align:middle line:90% Kevin? 00:23:19.620 --> 00:23:22.287 align:middle line:84% AUDIENCE: What's the slowest part of [INAUDIBLE]?? 00:23:22.287 --> 00:23:24.120 align:middle line:84% MATT HIRSCH: The slowest parts are the parts 00:23:24.120 --> 00:23:25.770 align:middle line:90% that I had to write. 00:23:25.770 --> 00:23:27.810 align:middle line:90% I mean, there are a lot tools. 00:23:27.810 --> 00:23:30.060 align:middle line:84% Like, OpenCV is a great tool for working 00:23:30.060 --> 00:23:32.347 align:middle line:90% with graphics in real time. 00:23:32.347 --> 00:23:33.930 align:middle line:84% There are Fourier transform libraries. 00:23:33.930 --> 00:23:36.540 align:middle line:90% 00:23:36.540 --> 00:23:39.880 align:middle line:84% Things that are slow on a modern computer are memory accesses. 00:23:39.880 --> 00:23:47.130 align:middle line:84% And so I end up having big sets of data that I have to remap. 00:23:47.130 --> 00:23:50.940 align:middle line:84% For example, you can measure a 2D data from the camera, right? 00:23:50.940 --> 00:23:52.770 align:middle line:84% But then you have to work with 4D data, 00:23:52.770 --> 00:23:54.600 align:middle line:84% and you have to remap it in a way 00:23:54.600 --> 00:23:57.970 align:middle line:84% that your Fourier transform library can understand it. 00:23:57.970 --> 00:23:59.978 align:middle line:84% And so that remapping is actually 00:23:59.978 --> 00:24:01.020 align:middle line:90% one of the longest steps. 00:24:01.020 --> 00:24:03.000 align:middle line:84% It's just simply reshuffling things. 00:24:03.000 --> 00:24:06.660 align:middle line:84% And I guess one of the most challenging practical problems 00:24:06.660 --> 00:24:09.727 align:middle line:84% here is literally just synchronizing everything. 00:24:09.727 --> 00:24:11.310 align:middle line:84% While computers are very fast, they're 00:24:11.310 --> 00:24:14.910 align:middle line:84% also not very reliable in terms of timing. 00:24:14.910 --> 00:24:17.490 align:middle line:84% So things can just happen whenever they happen. 00:24:17.490 --> 00:24:20.160 align:middle line:84% And especially with rendering things on a video card 00:24:20.160 --> 00:24:22.200 align:middle line:84% and trying to understand exactly when they're 00:24:22.200 --> 00:24:24.240 align:middle line:84% going to show up on a monitor, there 00:24:24.240 --> 00:24:26.700 align:middle line:84% are a lot of different and variable delays 00:24:26.700 --> 00:24:29.850 align:middle line:84% in that that are difficult to account for. 00:24:29.850 --> 00:24:33.360 align:middle line:84% So that's something I'm still working on. 00:24:33.360 --> 00:24:35.910 align:middle line:84% PROFESSOR: That's the kind of final project we want to see. 00:24:35.910 --> 00:24:37.620 align:middle line:90% AUDIENCE: [INAUDIBLE] question. 00:24:37.620 --> 00:24:41.760 align:middle line:84% Why is the diffuser a key part of this device? 00:24:41.760 --> 00:24:44.160 align:middle line:84% MATT HIRSCH: The diffuser is just our stand-in sensor. 00:24:44.160 --> 00:24:44.910 align:middle line:90% PROFESSOR: Yeah. 00:24:44.910 --> 00:24:47.010 align:middle line:84% I mean, this is the key part, right? 00:24:47.010 --> 00:24:51.060 align:middle line:84% The camera of the future will not look like anything 00:24:51.060 --> 00:24:51.930 align:middle line:90% like cameras today. 00:24:51.930 --> 00:24:55.110 align:middle line:84% Your LCD screen, a 15-inch screen, 00:24:55.110 --> 00:24:59.530 align:middle line:84% is actually your camera in the future. 00:24:59.530 --> 00:25:03.130 align:middle line:84% It's just that right now we don't have it. 00:25:03.130 --> 00:25:05.880 align:middle line:84% AUDIENCE: So you're using this as your sensor 00:25:05.880 --> 00:25:08.320 align:middle line:84% and then actually imaging the thing 00:25:08.320 --> 00:25:10.200 align:middle line:84% and then working from the image that you get. 00:25:10.200 --> 00:25:10.730 align:middle line:84% PROFESSOR: It's a shortcut for now. 00:25:10.730 --> 00:25:11.600 align:middle line:90% AUDIENCE: OK, I see. 00:25:11.600 --> 00:25:12.690 align:middle line:90% Yeah. 00:25:12.690 --> 00:25:14.820 align:middle line:84% PROFESSOR: But in the future, the whole thing 00:25:14.820 --> 00:25:15.610 align:middle line:90% will be a camera. 00:25:15.610 --> 00:25:17.790 align:middle line:84% So if somebody wants to take this concept further, 00:25:17.790 --> 00:25:19.710 align:middle line:84% by the way, what will you do when 00:25:19.710 --> 00:25:22.830 align:middle line:84% the camera is 15 inch wide, but only when 00:25:22.830 --> 00:25:24.180 align:middle line:90% you touch it you get an image? 00:25:24.180 --> 00:25:25.680 align:middle line:90% It's like a document scan. 00:25:25.680 --> 00:25:30.502 align:middle line:84% You take anything away from it, you just get a blurred thing. 00:25:30.502 --> 00:25:32.460 align:middle line:84% So if anyone wants to think about that further, 00:25:32.460 --> 00:25:33.490 align:middle line:90% I'll be very interested-- 00:25:33.490 --> 00:25:35.823 align:middle line:84% MATT HIRSCH: So they're selling these devices now that-- 00:25:35.823 --> 00:25:37.980 align:middle line:84% there's a laptop on sale in Japan 00:25:37.980 --> 00:25:42.090 align:middle line:84% that has a trackpad that's made from one of these optical LCDs. 00:25:42.090 --> 00:25:44.130 align:middle line:84% So this is a very near-term technology. 00:25:44.130 --> 00:25:46.000 align:middle line:84% In a couple of years, it'll be everywhere. 00:25:46.000 --> 00:25:49.033 align:middle line:84% So it's something cool to think about using. 00:25:49.033 --> 00:25:50.450 align:middle line:84% AUDIENCE: I have another question. 00:25:50.450 --> 00:25:52.290 align:middle line:90% What's the range of this device? 00:25:52.290 --> 00:25:56.750 align:middle line:84% Like, if I were to soften [INAUDIBLE].. 00:25:56.750 --> 00:25:59.332 align:middle line:84% MATT HIRSCH: Yeah, that's a good question. 00:25:59.332 --> 00:26:00.790 align:middle line:84% This is a very big parameter space. 00:26:00.790 --> 00:26:02.415 align:middle line:84% There are a lot of variables to change. 00:26:02.415 --> 00:26:04.000 align:middle line:84% So the prototype that we built, we 00:26:04.000 --> 00:26:05.832 align:middle line:84% optimized for about 50 centimeters 00:26:05.832 --> 00:26:06.790 align:middle line:90% in front of the screen. 00:26:06.790 --> 00:26:10.270 align:middle line:84% But you can think about basically changing 00:26:10.270 --> 00:26:12.490 align:middle line:84% that separation between the screen and the sensor 00:26:12.490 --> 00:26:15.370 align:middle line:84% and changing the pixel size of the screen and sensor. 00:26:15.370 --> 00:26:17.590 align:middle line:84% And all of those variables will allow 00:26:17.590 --> 00:26:20.350 align:middle line:84% you to control the range of depth that you can measure. 00:26:20.350 --> 00:26:22.970 align:middle line:84% PROFESSOR: And before we do it on a device like this 00:26:22.970 --> 00:26:24.700 align:middle line:90% and just do however and so on. 00:26:24.700 --> 00:26:27.680 align:middle line:84% That [INAUDIBLE] small form factor. 00:26:27.680 --> 00:26:30.180 align:middle line:84% MATT HIRSCH: Yeah, we just need to buy one of those laptops. 00:26:30.180 --> 00:26:30.847 align:middle line:90% PROFESSOR: Yeah. 00:26:30.847 --> 00:26:31.960 align:middle line:90% MATT HIRSCH: Yeah. 00:26:31.960 --> 00:26:33.400 align:middle line:84% PROFESSOR: Then the parameters will be very different, 00:26:33.400 --> 00:26:36.075 align:middle line:84% because then you don't expect to interact [INAUDIBLE].. 00:26:36.075 --> 00:26:41.530 align:middle line:90% 00:26:41.530 --> 00:26:46.350 align:middle line:84% So just to give some context for the class, of course, 00:26:46.350 --> 00:26:50.600 align:middle line:84% you did not do this whole thing in real time. 00:26:50.600 --> 00:26:52.997 align:middle line:84% You looked at only the capture, is that right? 00:26:52.997 --> 00:26:54.330 align:middle line:90% MATT HIRSCH: Yeah, that's right. 00:26:54.330 --> 00:26:58.890 align:middle line:84% And in fact, for the class, I started without an LCD 00:26:58.890 --> 00:27:01.660 align:middle line:84% because there are a lot of challenging 00:27:01.660 --> 00:27:05.210 align:middle line:84% hardware issues with getting an LCD to work like that. 00:27:05.210 --> 00:27:07.750 align:middle line:84% So what we started with was simply a printed mask. 00:27:07.750 --> 00:27:10.780 align:middle line:84% And there's a great resource right here in Cambridge 00:27:10.780 --> 00:27:14.360 align:middle line:84% called PageWorks, who can print very high-resolution masks. 00:27:14.360 --> 00:27:16.900 align:middle line:84% I think that's what you use for your [INAUDIBLE].. 00:27:16.900 --> 00:27:20.050 align:middle line:84% PROFESSOR: So he did a static version first, for the class, 00:27:20.050 --> 00:27:23.230 align:middle line:84% and did a proof of concept [INAUDIBLE] 00:27:23.230 --> 00:27:26.520 align:middle line:90% theory and the static prototype. 00:27:26.520 --> 00:27:29.310 align:middle line:84% And then, the two months after that, 00:27:29.310 --> 00:27:32.102 align:middle line:84% towards the SIGGRAPH deadline, he did all these things. 00:27:32.102 --> 00:27:33.810 align:middle line:84% Unfortunately, the paper was not accepted 00:27:33.810 --> 00:27:36.420 align:middle line:84% despite all the great work and all the results. 00:27:36.420 --> 00:27:38.422 align:middle line:84% That tells you how high the bar is. 00:27:38.422 --> 00:27:40.380 align:middle line:84% MATT HIRSCH: Luckily, we got to SIGGRAPH agent. 00:27:40.380 --> 00:27:45.155 align:middle line:84% So [INAUDIBLE] a school in Louisiana. 00:27:45.155 --> 00:27:47.280 align:middle line:84% AUDIENCE: Then you've got to buy the laptop, right? 00:27:47.280 --> 00:27:48.010 align:middle line:90% MATT HIRSCH: That's right. 00:27:48.010 --> 00:27:49.385 align:middle line:84% PROFESSOR: But it was cool enough 00:27:49.385 --> 00:27:51.660 align:middle line:84% that when he presented it at SIGGRAPH in New Orleans, 00:27:51.660 --> 00:27:55.320 align:middle line:84% he won the second best paper award [INAUDIBLE].. 00:27:55.320 --> 00:27:56.528 align:middle line:90% MATT HIRSCH: It was a poster. 00:27:56.528 --> 00:27:57.932 align:middle line:90% PROFESSOR: [INAUDIBLE]. 00:27:57.932 --> 00:28:00.050 align:middle line:90% So now he's a [INAUDIBLE]. 00:28:00.050 --> 00:28:02.690 align:middle line:84% MATT HIRSCH: I don't know what that means, but-- 00:28:02.690 --> 00:28:05.750 align:middle line:84% PROFESSOR: Eventually, [INAUDIBLE].. 00:28:05.750 --> 00:28:08.840 align:middle line:84% Cool, any other questions for Matt? 00:28:08.840 --> 00:28:12.930 align:middle line:84% So this is an example of how you think about image formation 00:28:12.930 --> 00:28:15.230 align:middle line:84% and image processing in higher dimensions. 00:28:15.230 --> 00:28:17.600 align:middle line:84% It opens up a completely new space. 00:28:17.600 --> 00:28:19.250 align:middle line:84% Don't think of camera as something 00:28:19.250 --> 00:28:22.250 align:middle line:84% that takes the 3D world and maps to the 2D image 00:28:22.250 --> 00:28:26.480 align:middle line:84% and all you can do is fiddle around with pixels. 00:28:26.480 --> 00:28:29.210 align:middle line:84% There's a lot more going on if you start thinking 00:28:29.210 --> 00:28:31.077 align:middle line:90% about the whole packet. 00:28:31.077 --> 00:28:32.660 align:middle line:84% So let's take a short break, and we'll 00:28:32.660 --> 00:28:36.110 align:middle line:84% come back and talk about other HCI 00:28:36.110 --> 00:28:38.780 align:middle line:84% applications of cameras in general, not just [INAUDIBLE] 00:28:38.780 --> 00:28:40.330 align:middle line:90% cameras. 00:28:40.330 --> 00:28:42.000 align:middle line:90%