WEBVTT 00:00:00.000 --> 00:00:02.490 align:middle line:84% The following content is provided under a Creative 00:00:02.490 --> 00:00:03.730 align:middle line:90% Commons license. 00:00:03.730 --> 00:00:06.120 align:middle line:84% Your support will help MIT OpenCourseWare 00:00:06.120 --> 00:00:10.060 align:middle line:84% continue to offer high-quality educational resources for free. 00:00:10.060 --> 00:00:12.720 align:middle line:84% To make a donation or to view additional materials 00:00:12.720 --> 00:00:16.650 align:middle line:84% from hundreds of MIT courses, visit MIT OpenCourseWare 00:00:16.650 --> 00:00:17.910 align:middle line:90% at ocw.mit.edu. 00:00:17.910 --> 00:00:20.222 align:middle line:90% 00:00:20.222 --> 00:00:21.930 align:middle line:84% RAMESH RASKAR: Kind of a summary of a lot 00:00:21.930 --> 00:00:23.050 align:middle line:90% of the things we talked about. 00:00:23.050 --> 00:00:24.550 align:middle line:84% So if you remember in the beginning, 00:00:24.550 --> 00:00:29.350 align:middle line:84% he said, you can just start with a pit. 00:00:29.350 --> 00:00:31.500 align:middle line:84% And then it kind of develops with the lens. 00:00:31.500 --> 00:00:34.620 align:middle line:84% But from even here, you can go down two different parts, 00:00:34.620 --> 00:00:39.390 align:middle line:84% either compound eyes, where each sensor or set of sensors 00:00:39.390 --> 00:00:45.820 align:middle line:84% have their own optics, like a sort of straw, or same lens-- 00:00:45.820 --> 00:00:49.170 align:middle line:84% sorry, the same pixel, might get image from multiple lenses, 00:00:49.170 --> 00:00:50.560 align:middle line:90% like here, right? 00:00:50.560 --> 00:00:52.410 align:middle line:90% So that's superposition. 00:00:52.410 --> 00:00:57.090 align:middle line:84% So this is a position, and this is superposition. 00:00:57.090 --> 00:01:00.840 align:middle line:84% And that concept of a position or superposition 00:01:00.840 --> 00:01:04.140 align:middle line:84% applies to all three types, shadows- or refraction- 00:01:04.140 --> 00:01:07.680 align:middle line:90% or reflection-based techniques. 00:01:07.680 --> 00:01:12.030 align:middle line:84% So we saw this last time, and we'll see how-- 00:01:12.030 --> 00:01:13.950 align:middle line:84% we already have some projects that are 00:01:13.950 --> 00:01:16.050 align:middle line:90% inspired by biological vision. 00:01:16.050 --> 00:01:17.730 align:middle line:84% You know, Matt is trying the chicken. 00:01:17.730 --> 00:01:21.150 align:middle line:90% And I think it's going to be-- 00:01:21.150 --> 00:01:23.340 align:middle line:90% [LAUGHTER] 00:01:23.340 --> 00:01:24.840 align:middle line:90% It is going to be very popular. 00:01:24.840 --> 00:01:28.920 align:middle line:84% And I believe Santiago-- where's Santiago? 00:01:28.920 --> 00:01:31.910 align:middle line:90% Oh, yeah, his triangle-- 00:01:31.910 --> 00:01:34.350 align:middle line:90% the piston, kind of-- 00:01:34.350 --> 00:01:36.580 align:middle line:90% so some really great ideas. 00:01:36.580 --> 00:01:38.550 align:middle line:84% So I'm glad a lot of these concepts 00:01:38.550 --> 00:01:40.540 align:middle line:84% are coming together in the final projects. 00:01:40.540 --> 00:01:44.370 align:middle line:84% So today, we'll talk about coded imaging. 00:01:44.370 --> 00:01:47.280 align:middle line:84% And the concept here is very simple, OK? 00:01:47.280 --> 00:01:49.590 align:middle line:84% So I'll start with this one, which 00:01:49.590 --> 00:01:55.470 align:middle line:84% is you have a taxi zipping very fast. 00:01:55.470 --> 00:01:58.380 align:middle line:84% And you want to kind of take a photo in such a way 00:01:58.380 --> 00:02:02.940 align:middle line:84% that you can recover the sharp detail afterwards in software. 00:02:02.940 --> 00:02:06.030 align:middle line:84% So it's a form of a co-design between how 00:02:06.030 --> 00:02:08.400 align:middle line:84% you capture the image and how you process the image. 00:02:08.400 --> 00:02:12.600 align:middle line:84% In a typical film camera, or even it is digital camera, 00:02:12.600 --> 00:02:16.060 align:middle line:84% you take the picture, and that's basically the end of the story. 00:02:16.060 --> 00:02:19.410 align:middle line:84% And here, you're trying to do something clever about how 00:02:19.410 --> 00:02:21.360 align:middle line:90% the picture is taken. 00:02:21.360 --> 00:02:24.540 align:middle line:84% So of course, there are other opportunities 00:02:24.540 --> 00:02:25.620 align:middle line:90% of capturing this. 00:02:25.620 --> 00:02:28.805 align:middle line:84% You can either take a really short exposure photo. 00:02:28.805 --> 00:02:30.180 align:middle line:84% But that's going to be very dark. 00:02:30.180 --> 00:02:33.930 align:middle line:84% If you take a high ISO, you can recover some information, 00:02:33.930 --> 00:02:35.760 align:middle line:90% but still quite dark. 00:02:35.760 --> 00:02:38.010 align:middle line:84% Or you can just take a long-exposure photo 00:02:38.010 --> 00:02:39.930 align:middle line:90% by keeping the shutter open. 00:02:39.930 --> 00:02:43.770 align:middle line:84% But then you will get a blurry photo, which is well exposed, 00:02:43.770 --> 00:02:45.930 align:middle line:84% but a lot of the high-frequency details are lost. 00:02:45.930 --> 00:02:49.902 align:middle line:84% And then if you try to apply some deblurring, 00:02:49.902 --> 00:02:51.610 align:middle line:84% you'll get a result that looks like this, 00:02:51.610 --> 00:02:52.777 align:middle line:90% which is kind of reasonable. 00:02:52.777 --> 00:02:57.555 align:middle line:84% You can see the number one on this, I guess, Thomas Train. 00:02:57.555 --> 00:03:00.630 align:middle line:90% 00:03:00.630 --> 00:03:02.950 align:middle line:84% But you get a lot of banding artifacts 00:03:02.950 --> 00:03:05.230 align:middle line:84% and a lot of repetition and noise here. 00:03:05.230 --> 00:03:07.200 align:middle line:84% AUDIENCE: So what are those lenses? 00:03:07.200 --> 00:03:08.242 align:middle line:90% RAMESH RASKAR: This lens? 00:03:08.242 --> 00:03:10.870 align:middle line:84% So when you try to recover this information, 00:03:10.870 --> 00:03:12.720 align:middle line:84% you start getting this banding artifacts. 00:03:12.720 --> 00:03:15.640 align:middle line:84% And we'll see it in the next slide, why that happens. 00:03:15.640 --> 00:03:21.750 align:middle line:84% So what's going on here is that if you have a sharp photo-- 00:03:21.750 --> 00:03:25.830 align:middle line:84% if you have a blurred photo, you can basically represent that 00:03:25.830 --> 00:03:29.490 align:middle line:84% as a sharp photo, where there is a convolution 00:03:29.490 --> 00:03:34.550 align:middle line:84% of the sharp photo with some kind of a convolution filter, 00:03:34.550 --> 00:03:35.050 align:middle line:90% OK? 00:03:35.050 --> 00:03:37.220 align:middle line:84% So if you look at-- where's my laser point? 00:03:37.220 --> 00:03:45.424 align:middle line:90% 00:03:45.424 --> 00:03:47.260 align:middle line:90% If you look at this-- 00:03:47.260 --> 00:03:52.090 align:middle line:84% the tip of letter one here, it's been 00:03:52.090 --> 00:03:55.150 align:middle line:84% blurred by a certain amount of pixels 00:03:55.150 --> 00:03:56.630 align:middle line:90% in the horizontal direction. 00:03:56.630 --> 00:03:58.880 align:middle line:84% And if you keep the shutter open for even longer, 00:03:58.880 --> 00:04:01.810 align:middle line:84% it'll blur correspondingly longer. 00:04:01.810 --> 00:04:04.540 align:middle line:84% So you have basically a 1D convolution 00:04:04.540 --> 00:04:07.540 align:middle line:84% that's converting this image into this image. 00:04:07.540 --> 00:04:09.610 align:middle line:84% And of course, the goal usually is 00:04:09.610 --> 00:04:11.620 align:middle line:84% this is a photo that you capture and you 00:04:11.620 --> 00:04:15.400 align:middle line:84% would like to invert and get back this photo. 00:04:15.400 --> 00:04:19.510 align:middle line:84% So one would say, OK, this converted 00:04:19.510 --> 00:04:20.810 align:middle line:90% with this gives me that. 00:04:20.810 --> 00:04:23.320 align:middle line:84% So just [INAUDIBLE] using the same filter 00:04:23.320 --> 00:04:25.420 align:middle line:90% and maybe you'll get that back. 00:04:25.420 --> 00:04:27.730 align:middle line:84% That doesn't work because something 00:04:27.730 --> 00:04:31.270 align:middle line:90% called division was 0. 00:04:31.270 --> 00:04:35.020 align:middle line:84% And the way to think about that is in the Fourier domain 00:04:35.020 --> 00:04:39.550 align:middle line:84% because convolution in the image domain, our primary domain, 00:04:39.550 --> 00:04:42.490 align:middle line:84% is multiplication in the Fourier-- 00:04:42.490 --> 00:04:45.740 align:middle line:90% just standard Fourier transform. 00:04:45.740 --> 00:04:49.930 align:middle line:84% So if you take the Fourier transform of this 00:04:49.930 --> 00:04:53.840 align:middle line:84% and multiply that by the Fourier transform of this, 00:04:53.840 --> 00:04:57.820 align:middle line:84% you will get the Fourier transform of this, OK? 00:04:57.820 --> 00:04:59.240 align:middle line:90% So let's say we take this photo. 00:04:59.240 --> 00:05:00.790 align:middle line:90% Find the Fourier transform here. 00:05:00.790 --> 00:05:05.230 align:middle line:84% Multiply that by the Fourier transform of a box 00:05:05.230 --> 00:05:06.950 align:middle line:90% function, which is a sync. 00:05:06.950 --> 00:05:09.040 align:middle line:84% So what that means is that I'm going 00:05:09.040 --> 00:05:11.740 align:middle line:84% to take the lowest frequency, multiply by that value. 00:05:11.740 --> 00:05:13.360 align:middle line:84% I'm going to take the next frequency, 00:05:13.360 --> 00:05:14.470 align:middle line:90% multiply it by this value. 00:05:14.470 --> 00:05:17.145 align:middle line:84% Next frequency, multiply by this value, and so on, right? 00:05:17.145 --> 00:05:18.520 align:middle line:84% We're just going to multiply each 00:05:18.520 --> 00:05:23.950 align:middle line:84% of the frequencies in the image by the amplitude of the Fourier 00:05:23.950 --> 00:05:25.660 align:middle line:90% transform of this. 00:05:25.660 --> 00:05:28.600 align:middle line:84% And you can already see that lower frequencies will 00:05:28.600 --> 00:05:31.000 align:middle line:84% be preserved, but higher frequencies 00:05:31.000 --> 00:05:33.775 align:middle line:90% will be highly attenuated. 00:05:33.775 --> 00:05:35.650 align:middle line:84% But there's also something strange happening. 00:05:35.650 --> 00:05:37.840 align:middle line:84% Even some of the lower frequencies 00:05:37.840 --> 00:05:43.000 align:middle line:84% are actually being set to 0, which means that in this photo, 00:05:43.000 --> 00:05:46.390 align:middle line:84% these frequencies are missing altogether. 00:05:46.390 --> 00:05:47.810 align:middle line:90% They have been suppressed. 00:05:47.810 --> 00:05:50.110 align:middle line:84% So it's not a traditional low pass filter. 00:05:50.110 --> 00:05:52.420 align:middle line:84% It's a low pass filter where some 00:05:52.420 --> 00:05:57.820 align:middle line:84% of the even lower frequencies are also being nullified, 00:05:57.820 --> 00:06:01.150 align:middle line:84% which means that if I tried to recover 00:06:01.150 --> 00:06:03.040 align:middle line:84% from this photo, this photo, there 00:06:03.040 --> 00:06:07.270 align:middle line:84% is no chance because I have already attenuated 00:06:07.270 --> 00:06:10.130 align:middle line:84% and have lost all those frequencies. 00:06:10.130 --> 00:06:13.510 align:middle line:84% So the moment you take the photo, the damage is done. 00:06:13.510 --> 00:06:16.720 align:middle line:84% And there's nothing you can do to recover those frequencies 00:06:16.720 --> 00:06:19.360 align:middle line:84% because in the Fourier domain, all 00:06:19.360 --> 00:06:21.550 align:middle line:84% you have to do is take the Fourier transform of this 00:06:21.550 --> 00:06:23.440 align:middle line:84% and divide by the Fourier transform 00:06:23.440 --> 00:06:24.760 align:middle line:90% of this, which is this. 00:06:24.760 --> 00:06:27.850 align:middle line:84% And it will give you the [INAUDIBLE],, OK? 00:06:27.850 --> 00:06:30.850 align:middle line:84% But the Fourier transform has some zeros, 00:06:30.850 --> 00:06:34.480 align:middle line:84% so you cannot divide those frequencies by 0 and recover 00:06:34.480 --> 00:06:36.290 align:middle line:90% an image. 00:06:36.290 --> 00:06:39.070 align:middle line:84% So the culprit here is really this box function, 00:06:39.070 --> 00:06:42.070 align:middle line:84% which is equivalent to-- when you release the shutter, 00:06:42.070 --> 00:06:44.230 align:middle line:90% opening the-- 00:06:44.230 --> 00:06:46.300 align:middle line:90% release your shutter button-- 00:06:46.300 --> 00:06:50.410 align:middle line:84% opening the shutter and keeping it open for exposure duration 00:06:50.410 --> 00:06:51.430 align:middle line:90% and closing it. 00:06:51.430 --> 00:06:53.840 align:middle line:84% But that's the most natural thing to do. 00:06:53.840 --> 00:06:57.680 align:middle line:84% But apparently, it's not the most effective. 00:06:57.680 --> 00:06:58.870 align:middle line:90% So what if you change that? 00:06:58.870 --> 00:07:03.230 align:middle line:90% 00:07:03.230 --> 00:07:04.490 align:middle line:90% What if you changed that? 00:07:04.490 --> 00:07:06.800 align:middle line:84% And instead of keeping the shutter open 00:07:06.800 --> 00:07:09.770 align:middle line:84% for the entire duration, you open and close it 00:07:09.770 --> 00:07:12.560 align:middle line:84% in a carefully chosen binary sequence. 00:07:12.560 --> 00:07:14.890 align:middle line:84% So for some time, the shutter was open, 00:07:14.890 --> 00:07:15.850 align:middle line:90% then shutter's closed. 00:07:15.850 --> 00:07:16.850 align:middle line:90% It's open for some time. 00:07:16.850 --> 00:07:17.750 align:middle line:90% Again, it's closed. 00:07:17.750 --> 00:07:20.370 align:middle line:84% Here, it's closed for quite some time, open for a short time. 00:07:20.370 --> 00:07:21.360 align:middle line:90% And so on. 00:07:21.360 --> 00:07:24.410 align:middle line:84% So at the end, you still get just one photo. 00:07:24.410 --> 00:07:28.950 align:middle line:84% But now something magical has happened because first of all, 00:07:28.950 --> 00:07:30.950 align:middle line:84% if you look at this number one, you'll 00:07:30.950 --> 00:07:33.570 align:middle line:84% see that it's not the same as before. 00:07:33.570 --> 00:07:35.885 align:middle line:84% It has-- it seems to have these replicas. 00:07:35.885 --> 00:07:39.260 align:middle line:90% 00:07:39.260 --> 00:07:42.050 align:middle line:84% And the reason why this is better 00:07:42.050 --> 00:07:44.330 align:middle line:84% is you take the Fourier transform of this. 00:07:44.330 --> 00:07:47.750 align:middle line:84% It's actually flat, which means it's preserving all 00:07:47.750 --> 00:07:50.190 align:middle line:90% the frequencies in the image. 00:07:50.190 --> 00:07:53.240 align:middle line:84% So we can be sure that, in this photo, 00:07:53.240 --> 00:07:55.650 align:middle line:84% all the spatial frequencies-- low frequencies, 00:07:55.650 --> 00:07:57.110 align:middle line:90% high frequencies-- 00:07:57.110 --> 00:07:58.380 align:middle line:90% they're all preserved. 00:07:58.380 --> 00:07:59.630 align:middle line:90% Of course, they're attenuated. 00:07:59.630 --> 00:08:01.460 align:middle line:90% It's not as high as-- 00:08:01.460 --> 00:08:02.540 align:middle line:90% it's not 1.0. 00:08:02.540 --> 00:08:03.530 align:middle line:90% It's reduced. 00:08:03.530 --> 00:08:05.240 align:middle line:90% Maybe it's 0.1 or so. 00:08:05.240 --> 00:08:07.460 align:middle line:84% So they're all attenuated, but there is still 00:08:07.460 --> 00:08:09.800 align:middle line:84% some hope to recover this photo back 00:08:09.800 --> 00:08:12.080 align:middle line:84% from this because, in the denominator, 00:08:12.080 --> 00:08:13.170 align:middle line:90% we will not have seen. 00:08:13.170 --> 00:08:15.770 align:middle line:90% 00:08:15.770 --> 00:08:18.980 align:middle line:84% So of course, if you try to implement this mechanically, 00:08:18.980 --> 00:08:22.250 align:middle line:84% where you open the shutter and then mechanically 00:08:22.250 --> 00:08:24.720 align:middle line:84% try to close the shutter, that will be problematic. 00:08:24.720 --> 00:08:28.070 align:middle line:84% So what we did was we used an LCD-- actually, 00:08:28.070 --> 00:08:29.540 align:middle line:90% a ferroelectric LCD-- 00:08:29.540 --> 00:08:32.150 align:middle line:84% that becomes opaque and transparent. 00:08:32.150 --> 00:08:35.299 align:middle line:84% And in the old virtual-reality screens 00:08:35.299 --> 00:08:39.830 align:middle line:84% or even some of the games, you have these eyeglasses 00:08:39.830 --> 00:08:45.860 align:middle line:84% that flicker at 60 Hertz for time sequentially 00:08:45.860 --> 00:08:47.665 align:middle line:84% so that you can see the left eye versus, 00:08:47.665 --> 00:08:50.270 align:middle line:84% like-- because they're the same glasses. 00:08:50.270 --> 00:08:53.690 align:middle line:84% And a traditional LCD, unfortunately, 00:08:53.690 --> 00:08:56.870 align:middle line:84% doesn't have a very high contrast. 00:08:56.870 --> 00:08:59.660 align:middle line:84% And Simon is discovering that one more time. 00:08:59.660 --> 00:09:04.730 align:middle line:84% But the ferroelectric LCDs have a contrast of 1,000 to 1. 00:09:04.730 --> 00:09:09.650 align:middle line:84% So when it's opaque, the amount of light that passes through, 00:09:09.650 --> 00:09:11.650 align:middle line:84% as compared to when it's transparent 00:09:11.650 --> 00:09:13.400 align:middle line:84% and the amount of light it passes through, 00:09:13.400 --> 00:09:15.800 align:middle line:90% the ratio is 1 of 2,000. 00:09:15.800 --> 00:09:20.300 align:middle line:84% So when you turn this ferroelectric LCD off, 00:09:20.300 --> 00:09:22.530 align:middle line:90% it's really, really opaque. 00:09:22.530 --> 00:09:23.030 align:middle line:90% Yeah. 00:09:23.030 --> 00:09:26.090 align:middle line:84% AUDIENCE: Couldn't you just do a high-speed video 00:09:26.090 --> 00:09:29.262 align:middle line:84% or just a [INAUDIBLE] taking video and put out your frames? 00:09:29.262 --> 00:09:30.720 align:middle line:84% RAMESH RASKAR: So the question was, 00:09:30.720 --> 00:09:32.550 align:middle line:84% why not just capture high-speed video 00:09:32.550 --> 00:09:37.570 align:middle line:84% and take all these frames, right, and then 00:09:37.570 --> 00:09:38.480 align:middle line:90% put them together? 00:09:38.480 --> 00:09:41.650 align:middle line:84% The problem is each of the frame will be extremely dark. 00:09:41.650 --> 00:09:44.380 align:middle line:84% So you are basically adding up a lot of noise. 00:09:44.380 --> 00:09:46.165 align:middle line:84% Every frame is dominated by noise. 00:09:46.165 --> 00:09:47.040 align:middle line:90% AUDIENCE: Yeah, yeah. 00:09:47.040 --> 00:09:50.040 align:middle line:90% 00:09:50.040 --> 00:09:54.940 align:middle line:84% RAMESH RASKAR: So when the shutter is transparent, 00:09:54.940 --> 00:09:55.840 align:middle line:90% that goes through. 00:09:55.840 --> 00:09:59.260 align:middle line:84% When the shutter is opaque, light doesn't go through. 00:09:59.260 --> 00:10:04.180 align:middle line:90% And that's your 1010 inquiry. 00:10:04.180 --> 00:10:06.500 align:middle line:84% So, again, the idea is very simple. 00:10:06.500 --> 00:10:10.690 align:middle line:84% Instead of keeping the shutter open for the entire duration 00:10:10.690 --> 00:10:12.730 align:middle line:84% and getting a well-exposed photo, 00:10:12.730 --> 00:10:17.545 align:middle line:84% the shutter is open for only half of the time. 00:10:17.545 --> 00:10:19.570 align:middle line:84% AUDIENCE: There is an issues there. 00:10:19.570 --> 00:10:22.850 align:middle line:84% The support for the representation of the Fourier 00:10:22.850 --> 00:10:25.060 align:middle line:84% domain of that function that you describe 00:10:25.060 --> 00:10:26.920 align:middle line:90% there is infinite, right? 00:10:26.920 --> 00:10:29.157 align:middle line:84% So you actually truncate this in order to-- 00:10:29.157 --> 00:10:31.240 align:middle line:84% RAMESH RASKAR: It's not infinite because you still 00:10:31.240 --> 00:10:33.376 align:middle line:90% have some width. 00:10:33.376 --> 00:10:37.930 align:middle line:84% AUDIENCE: Right, but you have infinite high frequencies there 00:10:37.930 --> 00:10:39.700 align:middle line:90% by the sharp conditions, right? 00:10:39.700 --> 00:10:41.200 align:middle line:84% RAMESH RASKAR: Yeah, you can think-- 00:10:41.200 --> 00:10:43.610 align:middle line:84% I mean, you can think of this one goes to infinity. 00:10:43.610 --> 00:10:45.980 align:middle line:84% But there's hardly any energy left. 00:10:45.980 --> 00:10:49.480 align:middle line:84% So although it goes to infinity, there is not much energy left. 00:10:49.480 --> 00:10:51.970 align:middle line:84% AUDIENCE: But when you get to invert the process then, 00:10:51.970 --> 00:10:55.615 align:middle line:84% that's why you're still not getting the perfect images to-- 00:10:55.615 --> 00:10:57.035 align:middle line:90% RAMESH RASKAR: In this case. 00:10:57.035 --> 00:10:58.410 align:middle line:84% AUDIENCE: --in this case as well. 00:10:58.410 --> 00:11:00.430 align:middle line:84% You still lost some high frequency, right? 00:11:00.430 --> 00:11:02.930 align:middle line:84% RAMESH RASKAR: So you haven't seen the results yet for this. 00:11:02.930 --> 00:11:04.750 align:middle line:90% AUDIENCE: You show the text. 00:11:04.750 --> 00:11:05.710 align:middle line:90% RAMESH RASKAR: Yes. 00:11:05.710 --> 00:11:07.990 align:middle line:84% So this is what it looks in this case. 00:11:07.990 --> 00:11:11.170 align:middle line:84% But it's a very controlled experiment in a laboratory. 00:11:11.170 --> 00:11:13.000 align:middle line:84% So you take the toy, and you move it 00:11:13.000 --> 00:11:15.380 align:middle line:90% in a very controlled way. 00:11:15.380 --> 00:11:17.630 align:middle line:84% And this is what you get in a traditional camera. 00:11:17.630 --> 00:11:19.630 align:middle line:84% And this is what you get in the flutter shutter. 00:11:19.630 --> 00:11:21.630 align:middle line:90% So these are real photos. 00:11:21.630 --> 00:11:22.630 align:middle line:90% And, yeah, you're right. 00:11:22.630 --> 00:11:24.220 align:middle line:84% I mean, you still get some noise. 00:11:24.220 --> 00:11:27.850 align:middle line:84% And actually, if you can pair this with ground truth, 00:11:27.850 --> 00:11:30.620 align:middle line:84% you'll see that it's OK, but it's not perfect. 00:11:30.620 --> 00:11:34.060 align:middle line:84% AUDIENCE: Yeah, so let's say that you took the 0s 00:11:34.060 --> 00:11:36.130 align:middle line:90% from the [INAUDIBLE], right? 00:11:36.130 --> 00:11:37.840 align:middle line:84% And you just replaced it by something 00:11:37.840 --> 00:11:41.200 align:middle line:84% that is pretty close to 0, but not 0. 00:11:41.200 --> 00:11:44.080 align:middle line:90% And if you invert the process-- 00:11:44.080 --> 00:11:46.360 align:middle line:84% RAMESH RASKAR: From here, this is what you get. 00:11:46.360 --> 00:11:47.620 align:middle line:90% AUDIENCE: That's-- OK-- 00:11:47.620 --> 00:11:49.210 align:middle line:84% RAMESH RASKAR: There's a deep learning of this. 00:11:49.210 --> 00:11:49.790 align:middle line:90% AUDIENCE: OK. 00:11:49.790 --> 00:11:53.440 align:middle line:84% RAMESH RASKAR: Yeah, and that's this loss of these frequencies 00:11:53.440 --> 00:11:58.860 align:middle line:84% also shows up as these artifacts at regular frequencies, 00:11:58.860 --> 00:12:00.580 align:middle line:90% at regular intervals. 00:12:00.580 --> 00:12:03.943 align:middle line:90% 00:12:03.943 --> 00:12:05.360 align:middle line:84% So, again, this one-- this doesn't 00:12:05.360 --> 00:12:07.160 align:middle line:90% go to infinity all the way. 00:12:07.160 --> 00:12:08.660 align:middle line:90% [INAUDIBLE] 00:12:08.660 --> 00:12:11.570 align:middle line:84% It cuts off and corresponding to the width. 00:12:11.570 --> 00:12:14.420 align:middle line:84% The width of this post was very short 00:12:14.420 --> 00:12:18.050 align:middle line:84% than yesterday were very far away. 00:12:18.050 --> 00:12:18.822 align:middle line:90% OK? 00:12:18.822 --> 00:12:19.322 align:middle line:90% Yeah? 00:12:19.322 --> 00:12:21.410 align:middle line:84% AUDIENCE: The filter is dependent from distance? 00:12:21.410 --> 00:12:23.660 align:middle line:84% RAMESH RASKAR: The filters depend on multiple factors. 00:12:23.660 --> 00:12:28.410 align:middle line:84% So if your toy is moving or your taxi's moving really slow, 00:12:28.410 --> 00:12:30.180 align:middle line:84% then there is no need to-- in this case, 00:12:30.180 --> 00:12:33.740 align:middle line:90% the sequence was about 51-- 00:12:33.740 --> 00:12:37.850 align:middle line:90% actually, 52 vector long. 00:12:37.850 --> 00:12:42.920 align:middle line:84% So let's say your exposure time is about 104 milliseconds. 00:12:42.920 --> 00:12:45.297 align:middle line:90% It's open for two milliseconds. 00:12:45.297 --> 00:12:46.880 align:middle line:84% Here, it's open for four milliseconds, 00:12:46.880 --> 00:12:49.850 align:middle line:84% off for two milliseconds, four milliseconds, two. 00:12:49.850 --> 00:12:53.010 align:middle line:84% Maybe it's off for eight milliseconds, two, and so on. 00:12:53.010 --> 00:12:54.360 align:middle line:90% AUDIENCE: Yes, but-- 00:12:54.360 --> 00:12:58.050 align:middle line:84% RAMESH RASKAR: But with a vector length of 52. 00:12:58.050 --> 00:13:00.172 align:middle line:84% AUDIENCE: This filter is in time? 00:13:00.172 --> 00:13:01.130 align:middle line:90% RAMESH RASKAR: In time. 00:13:01.130 --> 00:13:03.080 align:middle line:84% AUDIENCE: And you think about filter from space? 00:13:03.080 --> 00:13:04.280 align:middle line:84% RAMESH RASKAR: It corresponds automatically 00:13:04.280 --> 00:13:05.125 align:middle line:90% to filter in space. 00:13:05.125 --> 00:13:07.550 align:middle line:84% AUDIENCE: Yeah, so [INAUDIBLE] it's dependent on distance, 00:13:07.550 --> 00:13:07.940 align:middle line:90% if you-- 00:13:07.940 --> 00:13:08.870 align:middle line:84% RAMESH RASKAR: Yeah, the speed, you mean. 00:13:08.870 --> 00:13:10.400 align:middle line:90% AUDIENCE: --of faraway objects. 00:13:10.400 --> 00:13:11.780 align:middle line:90% RAMESH RASKAR: Yes. 00:13:11.780 --> 00:13:16.460 align:middle line:84% So your actual blur in the image may not be exactly 52 pixels. 00:13:16.460 --> 00:13:17.930 align:middle line:90% It might be 10 pixels. 00:13:17.930 --> 00:13:19.200 align:middle line:90% It could be 100 pixels. 00:13:19.200 --> 00:13:21.290 align:middle line:84% So your 52 vector is going to stretch 00:13:21.290 --> 00:13:24.260 align:middle line:84% or shrink based on how fast the object is moving. 00:13:24.260 --> 00:13:26.240 align:middle line:84% And you're saying that it also depends 00:13:26.240 --> 00:13:28.130 align:middle line:84% on how far the object is in space 00:13:28.130 --> 00:13:30.200 align:middle line:90% because faster-moving objects. 00:13:30.200 --> 00:13:33.230 align:middle line:84% And you mostly have to think about image space motion 00:13:33.230 --> 00:13:38.270 align:middle line:84% because the speed in the real world and-- the distance are 00:13:38.270 --> 00:13:39.680 align:middle line:90% they get-- 00:13:39.680 --> 00:13:42.480 align:middle line:84% you divide to normalize by the distance. 00:13:42.480 --> 00:13:45.530 align:middle line:84% So you only have to worry about the image space distance. 00:13:45.530 --> 00:13:48.168 align:middle line:90% Yeah. 00:13:48.168 --> 00:13:48.668 align:middle line:90% Go ahead. 00:13:48.668 --> 00:13:51.830 align:middle line:84% AUDIENCE: Could you get a similar effect if you had, 00:13:51.830 --> 00:13:56.840 align:middle line:84% like, instead of a hooded shutter, it could have flasher? 00:13:56.840 --> 00:13:59.720 align:middle line:90% RAMESH RASKAR: Yeah, exactly. 00:13:59.720 --> 00:14:04.130 align:middle line:84% So if you're in a dark room, you can just-- 00:14:04.130 --> 00:14:06.020 align:middle line:84% if you're in a dark room, then you 00:14:06.020 --> 00:14:08.510 align:middle line:84% can just strobe the light, rather than opening and closing 00:14:08.510 --> 00:14:10.720 align:middle line:90% the shutter. 00:14:10.720 --> 00:14:13.220 align:middle line:84% AUDIENCE: I think we might have a mobile demo of that scene. 00:14:13.220 --> 00:14:16.370 align:middle line:84% RAMESH RASKAR: [LAUGHS] Well, I don't know how fast you can-- 00:14:16.370 --> 00:14:18.690 align:middle line:84% AUDIENCE: Well, the problem is you can't [INAUDIBLE].. 00:14:18.690 --> 00:14:21.360 align:middle line:90% RAMESH RASKAR: Yeah. 00:14:21.360 --> 00:14:25.750 align:middle line:84% So what are some-- let's look at some pictures, actually. 00:14:25.750 --> 00:14:28.020 align:middle line:90% So here is a demo. 00:14:28.020 --> 00:14:31.410 align:middle line:84% I think I've shown it to you before. 00:14:31.410 --> 00:14:33.300 align:middle line:90% This is on Broadway. 00:14:33.300 --> 00:14:37.040 align:middle line:84% This women try to figure out the car make and the license plate 00:14:37.040 --> 00:14:37.540 align:middle line:90% number. 00:14:37.540 --> 00:14:39.800 align:middle line:90% What's the license plate number? 00:14:39.800 --> 00:14:40.642 align:middle line:90% AUDIENCE: 458. 00:14:40.642 --> 00:14:42.810 align:middle line:90% [INTERPOSING VOICES] 00:14:42.810 --> 00:14:45.690 align:middle line:90% AUDIENCE: 468. 00:14:45.690 --> 00:14:46.710 align:middle line:90% AUDIENCE: Something. 00:14:46.710 --> 00:14:48.530 align:middle line:90% RAMESH RASKAR: And the company? 00:14:48.530 --> 00:14:52.530 align:middle line:90% AUDIENCE: [? One more time. ?] 00:14:52.530 --> 00:14:53.640 align:middle line:90% RAMESH RASKAR: Yeah. 00:14:53.640 --> 00:14:58.033 align:middle line:84% So you get a reasonable result. But going back, 00:14:58.033 --> 00:14:59.700 align:middle line:84% what are the limitations of this method? 00:14:59.700 --> 00:15:04.010 align:middle line:90% 00:15:04.010 --> 00:15:04.518 align:middle line:90% Yes. 00:15:04.518 --> 00:15:06.060 align:middle line:84% AUDIENCE: You need to know the motion 00:15:06.060 --> 00:15:07.907 align:middle line:90% or the direction of the motion. 00:15:07.907 --> 00:15:10.490 align:middle line:84% RAMESH RASKAR: No, you need to know the point spread function, 00:15:10.490 --> 00:15:12.060 align:middle line:90% how the blur is created. 00:15:12.060 --> 00:15:15.350 align:middle line:84% If the car is moving from left to right versus right to left, 00:15:15.350 --> 00:15:18.500 align:middle line:84% you need to know that because the way your point spread 00:15:18.500 --> 00:15:22.445 align:middle line:84% function will be imposed on the scene will be different. 00:15:22.445 --> 00:15:24.165 align:middle line:84% AUDIENCE: You still inspect the lighting. 00:15:24.165 --> 00:15:26.790 align:middle line:84% RAMESH RASKAR: You just have the light-- very important, right? 00:15:26.790 --> 00:15:31.080 align:middle line:84% So this image is about half as bright as this one. 00:15:31.080 --> 00:15:33.950 align:middle line:90% What else? 00:15:33.950 --> 00:15:36.230 align:middle line:84% AUDIENCE: I guess there should be a little less 00:15:36.230 --> 00:15:37.995 align:middle line:90% of an acceleration of-- 00:15:37.995 --> 00:15:39.620 align:middle line:84% all of them should be moving the same-- 00:15:39.620 --> 00:15:40.578 align:middle line:90% RAMESH RASKAR: Exactly. 00:15:40.578 --> 00:15:44.840 align:middle line:84% So whatever is moving has to move at a constant speed. 00:15:44.840 --> 00:15:47.780 align:middle line:84% If we did 100 milliseconds, it picks up speed, 00:15:47.780 --> 00:15:50.570 align:middle line:84% then your assumption that the 52-length vector 00:15:50.570 --> 00:15:54.860 align:middle line:84% will map to some stretched or shrunk version of 52 00:15:54.860 --> 00:15:55.640 align:middle line:90% is not valid. 00:15:55.640 --> 00:15:57.750 align:middle line:84% Some parts will go faster and slower. 00:15:57.750 --> 00:15:58.250 align:middle line:90% What else? 00:15:58.250 --> 00:16:00.135 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:16:00.135 --> 00:16:01.010 align:middle line:90% RAMESH RASKAR: Sorry? 00:16:01.010 --> 00:16:02.660 align:middle line:84% AUDIENCE: If the object is moving 00:16:02.660 --> 00:16:06.843 align:middle line:84% in space, [INAUDIBLE] distance and then [INAUDIBLE].. 00:16:06.843 --> 00:16:08.510 align:middle line:84% RAMESH RASKAR: Yeah, so you-- so if it's 00:16:08.510 --> 00:16:10.760 align:middle line:84% moving in a perspective, for example, 00:16:10.760 --> 00:16:13.350 align:middle line:84% it's not so bad because you can rotate the image. 00:16:13.350 --> 00:16:14.600 align:middle line:90% And again, it'll become-- 00:16:14.600 --> 00:16:15.830 align:middle line:90% so that's not acceleration. 00:16:15.830 --> 00:16:17.750 align:middle line:90% That's still constant speed. 00:16:17.750 --> 00:16:19.610 align:middle line:84% It's acceleration in the measurement, 00:16:19.610 --> 00:16:22.070 align:middle line:84% but in the real world, it's still constant speed. 00:16:22.070 --> 00:16:23.810 align:middle line:84% So you can play with those tricks. 00:16:23.810 --> 00:16:25.610 align:middle line:84% You can either go to object space 00:16:25.610 --> 00:16:27.080 align:middle line:84% or you can come back to image space 00:16:27.080 --> 00:16:29.690 align:middle line:84% to make sure there is no acceleration. 00:16:29.690 --> 00:16:31.557 align:middle line:90% It's all linear. 00:16:31.557 --> 00:16:33.140 align:middle line:84% AUDIENCE: So does this technique still 00:16:33.140 --> 00:16:35.480 align:middle line:84% work if you're moving in multiple directions 00:16:35.480 --> 00:16:37.110 align:middle line:90% at once over the duration? 00:16:37.110 --> 00:16:40.040 align:middle line:84% RAMESH RASKAR: So if you have multiple cars, for example, 00:16:40.040 --> 00:16:41.798 align:middle line:84% and they're all independent, then 00:16:41.798 --> 00:16:44.090 align:middle line:84% it's fine because I can say this car is going this way. 00:16:44.090 --> 00:16:45.350 align:middle line:90% That car is going this way. 00:16:45.350 --> 00:16:47.058 align:middle line:84% As long as it's moving in a straight line 00:16:47.058 --> 00:16:49.380 align:middle line:90% at a constant speed, you're OK. 00:16:49.380 --> 00:16:54.020 align:middle line:84% But if the two cars overlap, what happens? 00:16:54.020 --> 00:16:56.625 align:middle line:90% Our model fails again. 00:16:56.625 --> 00:16:59.690 align:middle line:84% If two cars are partially overlapping 00:16:59.690 --> 00:17:02.690 align:middle line:84% during the exposure, it's possible, 00:17:02.690 --> 00:17:04.490 align:middle line:84% but it's more challenging because you 00:17:04.490 --> 00:17:08.760 align:middle line:84% don't know exactly how fast the two cars are moving. 00:17:08.760 --> 00:17:09.260 align:middle line:90% Yeah. 00:17:09.260 --> 00:17:12.319 align:middle line:84% AUDIENCE: Sorry, might we need to know 00:17:12.319 --> 00:17:15.240 align:middle line:84% how fast the car is moving when you're setting up your shutter? 00:17:15.240 --> 00:17:16.573 align:middle line:90% RAMESH RASKAR: No, when you're-- 00:17:16.573 --> 00:17:18.560 align:middle line:84% OK, so when you're setting up your shutter, 00:17:18.560 --> 00:17:20.750 align:middle line:84% if the car is moving really slow, 00:17:20.750 --> 00:17:23.869 align:middle line:84% and you don't expect it to blur by 52 pixels, 00:17:23.869 --> 00:17:26.329 align:middle line:84% and you expect it to blur by only 10 pixels, 00:17:26.329 --> 00:17:30.080 align:middle line:84% then using a 52 sequence is overkill. 00:17:30.080 --> 00:17:32.490 align:middle line:84% Maybe you should use a new sequence that's only about 10 00:17:32.490 --> 00:17:33.858 align:middle line:90% long or 11 long, right? 00:17:33.858 --> 00:17:34.650 align:middle line:90% So it's just like-- 00:17:34.650 --> 00:17:36.983 align:middle line:84% AUDIENCE: OK, but that's just so you can get more light. 00:17:36.983 --> 00:17:39.560 align:middle line:84% RAMESH RASKAR: No, that's so that it's most optimal for that 00:17:39.560 --> 00:17:40.737 align:middle line:90% setting, right? 00:17:40.737 --> 00:17:42.320 align:middle line:84% So it's like setting an exposure time. 00:17:42.320 --> 00:17:43.730 align:middle line:84% When I take a picture, the camera 00:17:43.730 --> 00:17:47.385 align:middle line:84% automatically decides what the exposure time should be. 00:17:47.385 --> 00:17:49.010 align:middle line:84% Similarly, you should look at the speed 00:17:49.010 --> 00:17:52.070 align:middle line:84% of how things are moving maybe with an ultrasound 00:17:52.070 --> 00:17:53.730 align:middle line:90% Doppler or whatever. 00:17:53.730 --> 00:17:55.512 align:middle line:84% And it says, things are not moving at all. 00:17:55.512 --> 00:17:57.470 align:middle line:84% So I should not use the flutter shutter at all. 00:17:57.470 --> 00:17:59.020 align:middle line:84% Until they're moving very slowly, 00:17:59.020 --> 00:18:00.710 align:middle line:84% maybe I should use a 10 long sequence. 00:18:00.710 --> 00:18:02.168 align:middle line:84% If things are moving a lot, maybe I 00:18:02.168 --> 00:18:04.580 align:middle line:90% should use a 52 sequence. 00:18:04.580 --> 00:18:07.490 align:middle line:84% And to answer your other question, where you need to do 00:18:07.490 --> 00:18:09.980 align:middle line:84% is when we solve the system, we need 00:18:09.980 --> 00:18:13.670 align:middle line:84% to know how long the blur is, which is true in other cases 00:18:13.670 --> 00:18:15.050 align:middle line:90% as well. 00:18:15.050 --> 00:18:16.940 align:middle line:84% You need to know how much the blur is. 00:18:16.940 --> 00:18:20.890 align:middle line:90% 00:18:20.890 --> 00:18:23.080 align:middle line:84% Another major disadvantage is let's 00:18:23.080 --> 00:18:26.510 align:middle line:90% say I want to take this bottle. 00:18:26.510 --> 00:18:29.770 align:middle line:84% And if I just rotate it and motion blur that, 00:18:29.770 --> 00:18:30.520 align:middle line:90% it will not work. 00:18:30.520 --> 00:18:32.950 align:middle line:84% For any point in the front that you're looking at it, 00:18:32.950 --> 00:18:34.070 align:middle line:90% it'll work. 00:18:34.070 --> 00:18:35.950 align:middle line:84% But the point that was in the back, 00:18:35.950 --> 00:18:39.100 align:middle line:84% that all of the 52 sequence-- maybe for the first 10, 00:18:39.100 --> 00:18:40.810 align:middle line:90% it was occluded. 00:18:40.810 --> 00:18:43.510 align:middle line:84% And the remaining 42, it was seen. 00:18:43.510 --> 00:18:48.460 align:middle line:84% You have to know exactly when that point became visible 00:18:48.460 --> 00:18:50.150 align:middle line:90% during that 52 window. 00:18:50.150 --> 00:18:53.680 align:middle line:84% So in general, the technique works well when 00:18:53.680 --> 00:18:57.280 align:middle line:90% things are moving naturally. 00:18:57.280 --> 00:19:01.120 align:middle line:84% But if somebody wants to do this kind of an experiment, 00:19:01.120 --> 00:19:04.990 align:middle line:84% or move things behind an occluder and move out, 00:19:04.990 --> 00:19:06.895 align:middle line:84% those are very challenging scenarios. 00:19:06.895 --> 00:19:08.980 align:middle line:84% AUDIENCE: Can you combine both horizontal 00:19:08.980 --> 00:19:11.597 align:middle line:90% and vertical [INAUDIBLE] masks? 00:19:11.597 --> 00:19:13.430 align:middle line:84% RAMESH RASKAR: Vertical, horizontal is fine. 00:19:13.430 --> 00:19:14.720 align:middle line:90% You can-- it doesn't matter. 00:19:14.720 --> 00:19:16.180 align:middle line:90% It could be moving vertically. 00:19:16.180 --> 00:19:18.730 align:middle line:84% Basically, your point spread function-- 00:19:18.730 --> 00:19:20.470 align:middle line:84% the blur function will be vertical rather 00:19:20.470 --> 00:19:21.190 align:middle line:90% than horizontal. 00:19:21.190 --> 00:19:24.585 align:middle line:84% AUDIENCE: Yeah, no, but if you have a combined motion, 00:19:24.585 --> 00:19:25.960 align:middle line:84% vertical and horizontal, you have 00:19:25.960 --> 00:19:27.160 align:middle line:90% to encode this with a mask? 00:19:27.160 --> 00:19:27.820 align:middle line:90% RAMESH RASKAR: No, no, no. 00:19:27.820 --> 00:19:29.050 align:middle line:90% So let's say the two cars-- 00:19:29.050 --> 00:19:29.770 align:middle line:90% one is moving-- 00:19:29.770 --> 00:19:31.630 align:middle line:84% AUDIENCE: [INAUDIBLE] diagonally from their [? English ?] way, 00:19:31.630 --> 00:19:31.840 align:middle line:90% right? 00:19:31.840 --> 00:19:32.965 align:middle line:90% RAMESH RASKAR: That's fine. 00:19:32.965 --> 00:19:35.440 align:middle line:84% As long as it's in any one direction, it's OK. 00:19:35.440 --> 00:19:37.275 align:middle line:90% So let me draw it. 00:19:37.275 --> 00:19:38.650 align:middle line:84% AUDIENCE: But if you take a sharp 00:19:38.650 --> 00:19:40.060 align:middle line:90% turn, you pass through, or-- 00:19:40.060 --> 00:19:41.268 align:middle line:90% RAMESH RASKAR: Yeah, exactly. 00:19:41.268 --> 00:19:43.110 align:middle line:84% So you have to assume that the point-- 00:19:43.110 --> 00:19:46.090 align:middle line:84% so the basic assumption is that if you 00:19:46.090 --> 00:19:50.480 align:middle line:84% take any point in the scene, it's moving in a straight line, 00:19:50.480 --> 00:19:51.477 align:middle line:90% let's say. 00:19:51.477 --> 00:19:53.560 align:middle line:84% And if you have an object, and every point of that 00:19:53.560 --> 00:19:55.720 align:middle line:84% object moves in a straight line, OK. 00:19:55.720 --> 00:19:57.778 align:middle line:84% It doesn't matter which direction and what speed. 00:19:57.778 --> 00:19:59.320 align:middle line:84% AUDIENCE: So this doesn't help at all 00:19:59.320 --> 00:20:01.903 align:middle line:84% with any image stabilization if somebody's holding the camera. 00:20:01.903 --> 00:20:03.237 align:middle line:90% RAMESH RASKAR: It helps as well. 00:20:03.237 --> 00:20:04.000 align:middle line:90% So if you have-- 00:20:04.000 --> 00:20:06.790 align:middle line:84% let's say you have a camera shape. 00:20:06.790 --> 00:20:09.490 align:middle line:84% And I take a picture of an LED, and it 00:20:09.490 --> 00:20:12.820 align:middle line:84% creates some curve like that because that kind of shape. 00:20:12.820 --> 00:20:15.970 align:middle line:84% If I know that curve, maybe I can put a gyro. 00:20:15.970 --> 00:20:19.280 align:middle line:84% Then I can, again, figure that out. 00:20:19.280 --> 00:20:22.600 align:middle line:84% So the problem here really is the point 00:20:22.600 --> 00:20:27.280 align:middle line:84% spread function or the blurred function is very critical. 00:20:27.280 --> 00:20:32.020 align:middle line:84% And this is what we want to study about half of the class. 00:20:32.020 --> 00:20:35.080 align:middle line:84% And the concept is very, very, very interesting 00:20:35.080 --> 00:20:37.360 align:middle line:90% because light is linear. 00:20:37.360 --> 00:20:38.920 align:middle line:90% So eventually, it's very linear. 00:20:38.920 --> 00:20:43.370 align:middle line:84% What happens to a point happens to the rest of the object. 00:20:43.370 --> 00:20:45.550 align:middle line:84% So if I have a car that's moving, 00:20:45.550 --> 00:20:49.000 align:middle line:84% and I tell you how exactly one point of the car 00:20:49.000 --> 00:20:52.600 align:middle line:84% is behaving in the image, I can tell you automatically 00:20:52.600 --> 00:20:55.660 align:middle line:84% how the rest of the car is behaving in the image 00:20:55.660 --> 00:20:57.310 align:middle line:84% because it's going to do-- all of it 00:20:57.310 --> 00:21:02.590 align:middle line:84% is going to have the same spread image. 00:21:02.590 --> 00:21:04.640 align:middle line:90% So you can either-- 00:21:04.640 --> 00:21:08.380 align:middle line:84% for experiments, you can just put an LED on the car 00:21:08.380 --> 00:21:10.420 align:middle line:90% and see how that LED moves. 00:21:10.420 --> 00:21:13.550 align:middle line:90% And that tells you everything. 00:21:13.550 --> 00:21:17.120 align:middle line:84% And I'm sure you use this trick in other scenarios 00:21:17.120 --> 00:21:23.083 align:middle line:84% where you look at a very small impulse 00:21:23.083 --> 00:21:24.250 align:middle line:90% and see how the response is. 00:21:24.250 --> 00:21:25.810 align:middle line:84% There's also an impulse response. 00:21:25.810 --> 00:21:27.730 align:middle line:84% For those of you in audio, you might 00:21:27.730 --> 00:21:32.020 align:middle line:84% want to check and see how the room [INAUDIBLE].. 00:21:32.020 --> 00:21:34.890 align:middle line:90% 00:21:34.890 --> 00:21:43.020 align:middle line:84% And when you're trying to find a speed of a car, [INAUDIBLE],, 00:21:43.020 --> 00:21:44.400 align:middle line:90% a very small impulse. 00:21:44.400 --> 00:21:46.540 align:middle line:90% And it answers and comes back. 00:21:46.540 --> 00:21:47.370 align:middle line:90% It does. 00:21:47.370 --> 00:21:55.657 align:middle line:84% The point spread function for your time of flight. 00:21:55.657 --> 00:21:56.990 align:middle line:90% So that's the same concept here. 00:21:56.990 --> 00:21:59.720 align:middle line:84% You just want to call leading the world, take a picture, 00:21:59.720 --> 00:22:01.910 align:middle line:90% and see how it works. 00:22:01.910 --> 00:22:04.880 align:middle line:84% And this whole field of order dimension 00:22:04.880 --> 00:22:10.920 align:middle line:84% is basically engineering of the point spread function. 00:22:10.920 --> 00:22:14.700 align:middle line:84% So if you take an ordinary camera, a film camera, 00:22:14.700 --> 00:22:17.370 align:middle line:84% and take a picture, you have no control 00:22:17.370 --> 00:22:21.630 align:middle line:84% over how light is spreading-- if something is moving 00:22:21.630 --> 00:22:25.170 align:middle line:84% or a focus has different color spectrum. 00:22:25.170 --> 00:22:29.100 align:middle line:84% And so an order dimension basically 00:22:29.100 --> 00:22:32.280 align:middle line:84% means you want to control how something 00:22:32.280 --> 00:22:34.990 align:middle line:90% is spreading on the image. 00:22:34.990 --> 00:22:38.170 align:middle line:84% So we're going to engineer activity of the camera. 00:22:38.170 --> 00:22:43.290 align:middle line:84% So in this particular case, a point that was moving 00:22:43.290 --> 00:22:45.640 align:middle line:90% created a blur like this. 00:22:45.640 --> 00:22:49.815 align:middle line:84% And by engineering the time point spread function, 00:22:49.815 --> 00:22:51.540 align:middle line:84% it stops looking a bit like that. 00:22:51.540 --> 00:22:55.300 align:middle line:84% It's going to look like that, all right? 00:22:55.300 --> 00:22:57.840 align:middle line:84% It's going to look like fashion [? wise. ?] 00:22:57.840 --> 00:23:00.990 align:middle line:84% And then it just turns out that this one is easier 00:23:00.990 --> 00:23:04.620 align:middle line:90% to deal with than this one. 00:23:04.620 --> 00:23:07.030 align:middle line:84% So that's the basic concept, engineering 00:23:07.030 --> 00:23:09.040 align:middle line:84% or actively changing the point spread function. 00:23:09.040 --> 00:23:12.830 align:middle line:90% 00:23:12.830 --> 00:23:17.030 align:middle line:84% So this is very counterintuitive because you would say, 00:23:17.030 --> 00:23:22.140 align:middle line:84% let me just build the best lens and the best exposure time. 00:23:22.140 --> 00:23:25.130 align:middle line:84% And so that kind of mimics the human eye. 00:23:25.130 --> 00:23:27.780 align:middle line:84% And once I have that, I have the best possible picture. 00:23:27.780 --> 00:23:31.010 align:middle line:84% But when it comes to actually extracting information 00:23:31.010 --> 00:23:34.730 align:middle line:84% from that scene, it turns out you 00:23:34.730 --> 00:23:40.460 align:middle line:84% need to strategically modify how the camera works 00:23:40.460 --> 00:23:43.310 align:middle line:84% so that all the information is somehow preserved. 00:23:43.310 --> 00:23:46.730 align:middle line:84% Now the problem is, even after you are very careful 00:23:46.730 --> 00:23:50.750 align:middle line:84% and you have captured that image, 00:23:50.750 --> 00:23:53.240 align:middle line:84% it's still going to be somewhat garbled. 00:23:53.240 --> 00:23:55.130 align:middle line:90% It's going to be mixed in. 00:23:55.130 --> 00:23:58.020 align:middle line:84% But that's where the co-design comes in. 00:23:58.020 --> 00:24:01.670 align:middle line:90% 00:24:01.670 --> 00:24:05.060 align:middle line:84% So once you have this image, there is some hope, 00:24:05.060 --> 00:24:06.710 align:middle line:84% there is some computational technique, 00:24:06.710 --> 00:24:10.380 align:middle line:84% that will allow you to go from here to here. 00:24:10.380 --> 00:24:13.280 align:middle line:84% And this is what kind of separates an animal 00:24:13.280 --> 00:24:17.780 align:middle line:84% eye from a computational eye because in most scenarios, 00:24:17.780 --> 00:24:20.120 align:middle line:84% an animal eye is just going to take the picture 00:24:20.120 --> 00:24:22.760 align:middle line:84% and try to make the best sense out of it. 00:24:22.760 --> 00:24:27.140 align:middle line:84% But a computational eye is going to apply a lot of processing 00:24:27.140 --> 00:24:29.330 align:middle line:84% to this and be able to recover that. 00:24:29.330 --> 00:24:33.650 align:middle line:84% As far as I know, animals don't have deconvolution circuitry 00:24:33.650 --> 00:24:35.330 align:middle line:90% or deep-learning circuitry. 00:24:35.330 --> 00:24:38.480 align:middle line:84% I can look at a blurry image and kind of figure out. 00:24:38.480 --> 00:24:40.400 align:middle line:84% I mean, this was a challenge for you, right? 00:24:40.400 --> 00:24:45.100 align:middle line:90% 00:24:45.100 --> 00:24:45.600 align:middle line:90% Right. 00:24:45.600 --> 00:24:47.640 align:middle line:84% So we have pretty sophisticated eyes, 00:24:47.640 --> 00:24:50.700 align:middle line:84% but we're still not able to deep learn what this is. 00:24:50.700 --> 00:24:53.730 align:middle line:84% If you have some prior knowledge of how the Volkswagen 00:24:53.730 --> 00:24:57.790 align:middle line:84% logo looks like, maybe you can say, OK, maybe that was this. 00:24:57.790 --> 00:24:59.970 align:middle line:84% But on the other hand, if I give you this, 00:24:59.970 --> 00:25:02.340 align:middle line:84% you're immediately willing to believe 00:25:02.340 --> 00:25:04.800 align:middle line:84% that this photo is a blurred version of this photo. 00:25:04.800 --> 00:25:07.400 align:middle line:90% 00:25:07.400 --> 00:25:09.620 align:middle line:84% And so kind of thinking about that 00:25:09.620 --> 00:25:14.390 align:middle line:84% is when you go from here to here, information is lost. 00:25:14.390 --> 00:25:16.130 align:middle line:84% When you go from here to here, we're 00:25:16.130 --> 00:25:18.990 align:middle line:84% trying to recover some information. 00:25:18.990 --> 00:25:22.130 align:middle line:84% So going from a sharp photo to a blurred photo is easy for us 00:25:22.130 --> 00:25:24.200 align:middle line:84% because we just have to lose some information 00:25:24.200 --> 00:25:27.650 align:middle line:84% or to imagine what it would look like if some of the information 00:25:27.650 --> 00:25:29.570 align:middle line:90% is removed from this image. 00:25:29.570 --> 00:25:34.450 align:middle line:90% 00:25:34.450 --> 00:25:38.160 align:middle line:84% So the goal of coded imaging is to come up 00:25:38.160 --> 00:25:41.550 align:middle line:84% with clever mechanisms so that we can capture light 00:25:41.550 --> 00:25:44.220 align:middle line:84% but not just by converting photons into electrons, 00:25:44.220 --> 00:25:46.560 align:middle line:84% but actually modulating those photons, 00:25:46.560 --> 00:25:48.390 align:middle line:84% either blocking them or attenuating them 00:25:48.390 --> 00:25:50.620 align:middle line:90% or bending them, and so on. 00:25:50.620 --> 00:25:53.550 align:middle line:84% So that that's why a computational camera is 00:25:53.550 --> 00:25:57.465 align:middle line:84% doing the computation not just in Silicon but also in optics. 00:25:57.465 --> 00:26:00.730 align:middle line:90% 00:26:00.730 --> 00:26:07.960 align:middle line:84% OK, so that was what we can do to preserve information 00:26:07.960 --> 00:26:10.868 align:middle line:90% in case of motion blur, right? 00:26:10.868 --> 00:26:12.410 align:middle line:84% And the circuit is very, very simple. 00:26:12.410 --> 00:26:16.990 align:middle line:84% You just take the hot shoe of the flash, and it triggers. 00:26:16.990 --> 00:26:19.210 align:middle line:84% When you lose the shutter, it triggers the circuit. 00:26:19.210 --> 00:26:23.590 align:middle line:84% And then you just cycle through the code that you care about. 00:26:23.590 --> 00:26:28.870 align:middle line:84% What can we do for defocus blur that is for motion blur? 00:26:28.870 --> 00:26:31.420 align:middle line:84% What can you do for defocus blur? 00:26:31.420 --> 00:26:35.328 align:middle line:84% We, again, want to engineer the point spread function. 00:26:35.328 --> 00:26:36.370 align:middle line:90% AUDIENCE: Spatial coding. 00:26:36.370 --> 00:26:37.690 align:middle line:90% RAMESH RASKAR: Spatial coding. 00:26:37.690 --> 00:26:39.355 align:middle line:84% How would you apply spatial coding? 00:26:39.355 --> 00:26:40.660 align:middle line:90% AUDIENCE: Coded aperture? 00:26:40.660 --> 00:26:41.440 align:middle line:90% RAMESH RASKAR: Coded aperture. 00:26:41.440 --> 00:26:43.855 align:middle line:84% So this is coded exposure, coded aperture-- very easy. 00:26:43.855 --> 00:26:46.520 align:middle line:90% 00:26:46.520 --> 00:26:49.000 align:middle line:84% And all you're going to do is put some kind of a code 00:26:49.000 --> 00:26:51.320 align:middle line:90% in the aperture of the lens. 00:26:51.320 --> 00:26:53.545 align:middle line:84% And this is how, actually, it started 00:26:53.545 --> 00:26:56.090 align:middle line:84% in the days of-- in scientific imaging, 00:26:56.090 --> 00:26:59.620 align:middle line:84% especially in astronomy, coded apertures are very well known. 00:26:59.620 --> 00:27:03.190 align:middle line:84% And those of you attended Professor Han's lecture 00:27:03.190 --> 00:27:08.120 align:middle line:84% on Wednesday, that's what he talked about, coded apertures. 00:27:08.120 --> 00:27:10.730 align:middle line:84% So I've been following this for a long, long time. 00:27:10.730 --> 00:27:15.430 align:middle line:84% And I thought, it must be useful for something in photography. 00:27:15.430 --> 00:27:19.190 align:middle line:84% And so I said, OK, let's try to put a coded aperture 00:27:19.190 --> 00:27:25.550 align:middle line:84% in the camera and see if we can deal with focus and so on. 00:27:25.550 --> 00:27:28.180 align:middle line:90% And that was back in 2004. 00:27:28.180 --> 00:27:31.150 align:middle line:84% And we tried it for six months, and it just didn't work. 00:27:31.150 --> 00:27:35.300 align:middle line:84% It was really frustrating-- really, really frustrating. 00:27:35.300 --> 00:27:37.630 align:middle line:84% And then one fine day, I said, OK, 00:27:37.630 --> 00:27:41.020 align:middle line:84% if you can do this in space, I'm sure we can do this in time 00:27:41.020 --> 00:27:42.380 align:middle line:90% as well. 00:27:42.380 --> 00:27:44.890 align:middle line:84% And so we did this, and this worked right away, 00:27:44.890 --> 00:27:46.270 align:middle line:90% within a couple of weeks. 00:27:46.270 --> 00:27:49.360 align:middle line:84% So we went ahead and built this whole system. 00:27:49.360 --> 00:27:51.070 align:middle line:90% And that was just a graph paper. 00:27:51.070 --> 00:27:53.703 align:middle line:84% And then we said, OK, let's come back and think about this. 00:27:53.703 --> 00:27:54.370 align:middle line:90% What's going on? 00:27:54.370 --> 00:27:57.320 align:middle line:90% Why don't we get good results? 00:27:57.320 --> 00:27:59.950 align:middle line:84% So it took almost two years to realize 00:27:59.950 --> 00:28:05.350 align:middle line:84% that to put this coded aperture in a camera, 00:28:05.350 --> 00:28:07.300 align:middle line:84% there are only a few places where you 00:28:07.300 --> 00:28:09.470 align:middle line:90% can put it to get good results. 00:28:09.470 --> 00:28:13.700 align:middle line:84% So out of that came this particular experiment. 00:28:13.700 --> 00:28:17.500 align:middle line:84% So I have a colleague, Jim Kobler, at MG Edge. 00:28:17.500 --> 00:28:21.970 align:middle line:84% And one day, he showed me-- this is his lens, by the way. 00:28:21.970 --> 00:28:28.000 align:middle line:84% He was telling me the story that he was fishing with his camera, 00:28:28.000 --> 00:28:31.900 align:middle line:84% and some creature came out of the water, 00:28:31.900 --> 00:28:33.220 align:middle line:90% some kind of an alligator. 00:28:33.220 --> 00:28:38.620 align:middle line:84% And he lost his balance, and the boat flipped upside down. 00:28:38.620 --> 00:28:41.710 align:middle line:84% Somehow, he managed to flip back in. 00:28:41.710 --> 00:28:43.510 align:middle line:90% And the alligator went away. 00:28:43.510 --> 00:28:46.300 align:middle line:84% But it completely damaged his camera that was with him, 00:28:46.300 --> 00:28:47.800 align:middle line:90% and it just wouldn't work. 00:28:47.800 --> 00:28:52.210 align:middle line:84% So he just took out his lens, which is a standard Canon lens. 00:28:52.210 --> 00:28:54.790 align:middle line:84% And he said, let's open it all the way. 00:28:54.790 --> 00:28:58.000 align:middle line:84% So he ripped open all the damage. 00:28:58.000 --> 00:29:01.670 align:middle line:84% It had all the mud in it and so on. 00:29:01.670 --> 00:29:05.350 align:middle line:84% And then he just showed me this thing as is. 00:29:05.350 --> 00:29:09.790 align:middle line:84% And it was very fascinating because this is a standard film 00:29:09.790 --> 00:29:13.390 align:middle line:84% lens, which, of course, can also be used with a digital camera. 00:29:13.390 --> 00:29:16.270 align:middle line:84% And this is a fixed focal length lens. 00:29:16.270 --> 00:29:20.120 align:middle line:84% It's 100-millimeter focal length lens. 00:29:20.120 --> 00:29:24.213 align:middle line:84% And when you focus with this, it works in very interesting ways. 00:29:24.213 --> 00:29:26.380 align:middle line:84% First of all, it doesn't have a single lens element. 00:29:26.380 --> 00:29:28.280 align:middle line:90% It has multiple lens elements. 00:29:28.280 --> 00:29:30.340 align:middle line:84% So when you change the focus, it has 00:29:30.340 --> 00:29:32.800 align:middle line:84% to do some really interesting things. 00:29:32.800 --> 00:29:37.150 align:middle line:84% It has to deal with chromatic aberration, 00:29:37.150 --> 00:29:41.300 align:middle line:84% geometric aberrations, such as radial distortion, and so on. 00:29:41.300 --> 00:29:43.720 align:middle line:84% So it has to move all these lenses 00:29:43.720 --> 00:29:46.810 align:middle line:90% with corresponding ratios, OK? 00:29:46.810 --> 00:29:49.150 align:middle line:84% So I'll pass this around, and you'll 00:29:49.150 --> 00:29:53.230 align:middle line:84% see that there are these notches on this lens that 00:29:53.230 --> 00:29:55.240 align:middle line:90% are in a parabolic fashion. 00:29:55.240 --> 00:29:59.242 align:middle line:84% So when I wrote this, the internal lens-- 00:29:59.242 --> 00:30:01.450 align:middle line:84% the outermost lens and the innermost lens instruments 00:30:01.450 --> 00:30:02.290 align:middle line:90% are the same place. 00:30:02.290 --> 00:30:06.550 align:middle line:84% But all the inner lenses move with some particular ratio. 00:30:06.550 --> 00:30:10.330 align:middle line:84% It's amazing the way it's structured, right? 00:30:10.330 --> 00:30:13.420 align:middle line:84% So the multiple lenses are moving every time I move this. 00:30:13.420 --> 00:30:14.860 align:middle line:84% And they're moving because they're 00:30:14.860 --> 00:30:17.740 align:middle line:90% guided through these groups. 00:30:17.740 --> 00:30:19.810 align:middle line:84% But there's one particular location 00:30:19.810 --> 00:30:24.900 align:middle line:84% that does not change in this lens, and that's the aperture. 00:30:24.900 --> 00:30:27.210 align:middle line:84% So we said, let's look at this aperture. 00:30:27.210 --> 00:30:31.650 align:middle line:84% And back then, it was still a reasonable-looking lens. 00:30:31.650 --> 00:30:36.530 align:middle line:84% So we went in our lab, and we cut open all the way. 00:30:36.530 --> 00:30:40.330 align:middle line:84% And you can start putting new apertures in this plane. 00:30:40.330 --> 00:30:46.530 align:middle line:84% So you can cut open that particular guy 00:30:46.530 --> 00:30:47.950 align:middle line:90% and start putting this aperture. 00:30:47.950 --> 00:30:53.430 align:middle line:84% Now it turns out the center of production of this lens 00:30:53.430 --> 00:30:57.180 align:middle line:84% is very carefully designed by camera makers 00:30:57.180 --> 00:30:59.910 align:middle line:84% to be the same plane where you put your aperture. 00:30:59.910 --> 00:31:03.180 align:middle line:84% So when you change your f-stop and decrease it 00:31:03.180 --> 00:31:04.890 align:middle line:84% and increase it, it's all happening 00:31:04.890 --> 00:31:07.170 align:middle line:90% in the center of projection. 00:31:07.170 --> 00:31:10.020 align:middle line:84% Everybody knows central projection? 00:31:10.020 --> 00:31:13.380 align:middle line:84% So when you think about a visual camera, 00:31:13.380 --> 00:31:15.930 align:middle line:84% you make this very simplistic assumption. 00:31:15.930 --> 00:31:18.420 align:middle line:84% That is a pinhole, and there's a sensor. 00:31:18.420 --> 00:31:20.610 align:middle line:84% And when you put a lens, we assume 00:31:20.610 --> 00:31:22.860 align:middle line:84% that the center of the lens is the central projection, 00:31:22.860 --> 00:31:28.320 align:middle line:84% that this always can be assumed to go to that point. 00:31:28.320 --> 00:31:35.093 align:middle line:84% When you have a bunch of lenses, like way over here, 00:31:35.093 --> 00:31:36.635 align:middle line:84% where is the center of protectionism? 00:31:36.635 --> 00:31:39.980 align:middle line:84% Is it here or here or here or here? 00:31:39.980 --> 00:31:41.840 align:middle line:84% And of course, there is-- you can 00:31:41.840 --> 00:31:44.850 align:middle line:84% take a collection of these lenses 00:31:44.850 --> 00:31:49.280 align:middle line:84% and create one single center of projection for normal cameras. 00:31:49.280 --> 00:31:51.220 align:middle line:84% For professional lenses, that's not true. 00:31:51.220 --> 00:31:55.040 align:middle line:84% But for normal cameras, you have the central projection. 00:31:55.040 --> 00:31:58.160 align:middle line:84% But again, conceptually assume that all the rays 00:31:58.160 --> 00:32:00.200 align:middle line:84% are going through that point because you 00:32:00.200 --> 00:32:02.840 align:middle line:84% can replace this whole thing by one 00:32:02.840 --> 00:32:06.420 align:middle line:90% single lens in a [INAUDIBLE]. 00:32:06.420 --> 00:32:12.300 align:middle line:84% So finding that plane is actually a tricky problem. 00:32:12.300 --> 00:32:15.110 align:middle line:84% And in retrospect, it's very easy. 00:32:15.110 --> 00:32:18.030 align:middle line:84% If the lens makers are putting everything there, 00:32:18.030 --> 00:32:19.950 align:middle line:84% we should put a recorded aperture also 00:32:19.950 --> 00:32:20.700 align:middle line:90% in the same plane. 00:32:20.700 --> 00:32:22.867 align:middle line:84% So initially we said, oh, let's put it in the front. 00:32:22.867 --> 00:32:23.910 align:middle line:90% Let's put it in the back. 00:32:23.910 --> 00:32:25.150 align:middle line:90% We tried all those things. 00:32:25.150 --> 00:32:27.570 align:middle line:84% But that creates a blur that's not constant 00:32:27.570 --> 00:32:28.500 align:middle line:90% all over the image. 00:32:28.500 --> 00:32:29.880 align:middle line:90% And it has a lot of issues. 00:32:29.880 --> 00:32:32.010 align:middle line:84% But placing it over there, it turns out 00:32:32.010 --> 00:32:33.600 align:middle line:90% you get the same blur. 00:32:33.600 --> 00:32:36.000 align:middle line:84% So what exactly happens if you take 00:32:36.000 --> 00:32:39.990 align:middle line:84% a picture of a point light, and everything is a sharp focus? 00:32:39.990 --> 00:32:44.530 align:middle line:90% Nothing changes, OK? 00:32:44.530 --> 00:32:46.870 align:middle line:84% If you have just an open aperture 00:32:46.870 --> 00:32:51.810 align:middle line:84% and take a picture of a point light, it looks like a disc. 00:32:51.810 --> 00:32:58.530 align:middle line:84% Now what's going to happen when you put this code, like the 7 00:32:58.530 --> 00:33:01.710 align:middle line:84% by 7 mask, and take a out-of-focus picture? 00:33:01.710 --> 00:33:03.392 align:middle line:90% What will happen to the LED? 00:33:03.392 --> 00:33:05.100 align:middle line:84% AUDIENCE: It's going to look like a code. 00:33:05.100 --> 00:33:08.400 align:middle line:84% RAMESH RASKAR: It's going to look like the code, right? 00:33:08.400 --> 00:33:10.450 align:middle line:84% And why is that-- why is that happening? 00:33:10.450 --> 00:33:12.390 align:middle line:84% So let's think about [INAUDIBLE] focus. 00:33:12.390 --> 00:33:17.740 align:middle line:90% 00:33:17.740 --> 00:33:21.670 align:middle line:90% So we have our lens, right? 00:33:21.670 --> 00:33:23.590 align:middle line:90% And we have a point light. 00:33:23.590 --> 00:33:25.060 align:middle line:90% And we will put some code here. 00:33:25.060 --> 00:33:27.760 align:middle line:90% 00:33:27.760 --> 00:33:31.580 align:middle line:84% When it's in sharp focus, it doesn't really 00:33:31.580 --> 00:33:34.212 align:middle line:90% matter what the code is. 00:33:34.212 --> 00:33:36.170 align:middle line:84% Basically, you're talking about half the light, 00:33:36.170 --> 00:33:37.820 align:middle line:84% so the photo will be half a square. 00:33:37.820 --> 00:33:40.610 align:middle line:84% But other than that, it looks like an ordinary focus. 00:33:40.610 --> 00:33:43.890 align:middle line:84% And that's why if you have some dust on your lens 00:33:43.890 --> 00:33:45.800 align:middle line:84% and so on, usually it doesn't matter 00:33:45.800 --> 00:33:53.270 align:middle line:84% unless you have the dust all the way on your front lens 00:33:53.270 --> 00:33:56.330 align:middle line:84% because the center operation's over here. 00:33:56.330 --> 00:33:59.160 align:middle line:84% So if the dust was over here, nothing will happen. 00:33:59.160 --> 00:34:01.490 align:middle line:84% The image will be slightly darker. 00:34:01.490 --> 00:34:03.170 align:middle line:84% But if the dust is all in the front, 00:34:03.170 --> 00:34:04.700 align:middle line:90% then you start seeing distance. 00:34:04.700 --> 00:34:09.500 align:middle line:84% Anyway, so when it's in sharp focus, you just see the point. 00:34:09.500 --> 00:34:17.310 align:middle line:84% But let's say that your autofocus here. 00:34:17.310 --> 00:34:18.060 align:middle line:90% What will you see? 00:34:18.060 --> 00:34:19.768 align:middle line:84% You will see the same exact [INAUDIBLE].. 00:34:19.768 --> 00:34:21.540 align:middle line:90% So the [? ray ?] comes in. 00:34:21.540 --> 00:34:23.460 align:middle line:90% It's blocked. 00:34:23.460 --> 00:34:24.239 align:middle line:90% This ray goes in. 00:34:24.239 --> 00:34:25.679 align:middle line:90% It goes through. 00:34:25.679 --> 00:34:26.429 align:middle line:90% This ray comes in. 00:34:26.429 --> 00:34:27.900 align:middle line:90% It's blocked. 00:34:27.900 --> 00:34:29.580 align:middle line:84% This ray goes through, and so on. 00:34:29.580 --> 00:34:31.455 align:middle line:84% So basically, you'll see the same [? boat. ?] 00:34:31.455 --> 00:34:33.326 align:middle line:84% If you put the sensor all the way here, 00:34:33.326 --> 00:34:34.409 align:middle line:90% you'll see the whole code. 00:34:34.409 --> 00:34:38.100 align:middle line:84% If you start moving away, the code will shrink. 00:34:38.100 --> 00:34:42.300 align:middle line:84% And eventually, when you put it here, we get another code. 00:34:42.300 --> 00:34:44.440 align:middle line:84% That's exactly what's happening here. 00:34:44.440 --> 00:34:48.150 align:middle line:84% When it's auto focus, we just see the code, all right? 00:34:48.150 --> 00:34:51.940 align:middle line:84% By the way, this is the same idea behind another project, 00:34:51.940 --> 00:34:53.699 align:middle line:90% which is [INAUDIBLE]. 00:34:53.699 --> 00:34:55.739 align:middle line:84% So the idea came around at the same time 00:34:55.739 --> 00:34:57.150 align:middle line:90% of how to make this happen. 00:34:57.150 --> 00:34:59.990 align:middle line:90% 00:34:59.990 --> 00:35:00.500 align:middle line:90% OK. 00:35:00.500 --> 00:35:04.700 align:middle line:84% AUDIENCE: Wouldn't the imaging of this code now still 00:35:04.700 --> 00:35:06.470 align:middle line:90% have to be blurred? 00:35:06.470 --> 00:35:08.540 align:middle line:84% Like, so is that basically multiple 00:35:08.540 --> 00:35:09.900 align:middle line:90% apertures that you're seeing? 00:35:09.900 --> 00:35:14.390 align:middle line:84% RAMESH RASKAR: Yeah, so the photo here is nothing-- 00:35:14.390 --> 00:35:18.650 align:middle line:84% the photo here that you see is still blurred. 00:35:18.650 --> 00:35:22.720 align:middle line:84% It's just that it's blurred in a slightly different way-- 00:35:22.720 --> 00:35:24.350 align:middle line:90% strange. 00:35:24.350 --> 00:35:26.840 align:middle line:84% Here, it's blurred with that shape. 00:35:26.840 --> 00:35:29.690 align:middle line:84% Every point is blurred with that spread function. 00:35:29.690 --> 00:35:32.120 align:middle line:84% And you cannot see anything on the resolution chart. 00:35:32.120 --> 00:35:35.720 align:middle line:84% But here, if I just promote this guy-- 00:35:35.720 --> 00:35:40.930 align:middle line:84% no, that won't work because I'm in a different mode. 00:35:40.930 --> 00:35:45.770 align:middle line:90% 00:35:45.770 --> 00:35:48.870 align:middle line:84% If I look at this picture, you will see that-- 00:35:48.870 --> 00:35:50.170 align:middle line:90% so this is a sharp photo. 00:35:50.170 --> 00:35:52.240 align:middle line:90% It's blurred with disc. 00:35:52.240 --> 00:35:55.390 align:middle line:84% And it's blurred with that function. 00:35:55.390 --> 00:35:58.030 align:middle line:84% You can already see that it seems to preserve 00:35:58.030 --> 00:35:59.260 align:middle line:90% slightly more information. 00:35:59.260 --> 00:36:01.677 align:middle line:84% But it's still-- you won't be able to with your naked eye. 00:36:01.677 --> 00:36:05.260 align:middle line:84% You'll not be able to figure out what underlying patterns are. 00:36:05.260 --> 00:36:08.650 align:middle line:84% But it turns out, after the blurring, you can. 00:36:08.650 --> 00:36:10.750 align:middle line:84% All right, so then you can do these simple tricks, 00:36:10.750 --> 00:36:15.470 align:middle line:84% where the person you're interested in is out of focus. 00:36:15.470 --> 00:36:18.730 align:middle line:84% But then you can refocus digitally. 00:36:18.730 --> 00:36:22.630 align:middle line:84% So this is the input photo and the stock photos, all right? 00:36:22.630 --> 00:36:27.070 align:middle line:84% So the same exact trick, which is in case of motion, 00:36:27.070 --> 00:36:29.380 align:middle line:84% we created a point spread function 00:36:29.380 --> 00:36:33.040 align:middle line:84% that was engineered in one dimension. 00:36:33.040 --> 00:36:35.920 align:middle line:84% And here, we are engineering a point spread function 00:36:35.920 --> 00:36:37.870 align:middle line:90% that's two dimensional. 00:36:37.870 --> 00:36:43.130 align:middle line:84% So here, we know that the Fourier transform of this 1D 00:36:43.130 --> 00:36:45.910 align:middle line:90% 52-length vector is broadband. 00:36:45.910 --> 00:36:47.470 align:middle line:84% It has energy at all the frequency. 00:36:47.470 --> 00:36:48.745 align:middle line:90% What can we say about this? 00:36:48.745 --> 00:36:51.202 align:middle line:90% 00:36:51.202 --> 00:36:52.160 align:middle line:90% It's fully transformed. 00:36:52.160 --> 00:36:55.250 align:middle line:90% What can we say? 00:36:55.250 --> 00:36:57.020 align:middle line:90% It's still 7 by 7. 00:36:57.020 --> 00:36:59.945 align:middle line:84% So its Fourier transform is also 7 by 7. 00:36:59.945 --> 00:37:03.260 align:middle line:84% When it's 52, its Fourier transform is 52 long. 00:37:03.260 --> 00:37:08.390 align:middle line:90% 00:37:08.390 --> 00:37:10.925 align:middle line:84% AUDIENCE: It's more distributed instead of just all being 00:37:10.925 --> 00:37:11.840 align:middle line:90% near the center. 00:37:11.840 --> 00:37:15.020 align:middle line:84% RAMESH RASKAR: So in 1D, this is what we saw, right? 00:37:15.020 --> 00:37:17.340 align:middle line:90% Its Fourier transform is flat. 00:37:17.340 --> 00:37:20.570 align:middle line:84% So there are 52 entries here, and almost all of them 00:37:20.570 --> 00:37:23.700 align:middle line:90% are the same. 00:37:23.700 --> 00:37:26.235 align:middle line:84% Now we're saying, think about the problem in 2D. 00:37:26.235 --> 00:37:31.360 align:middle line:90% 00:37:31.360 --> 00:37:35.230 align:middle line:84% And what's the Fourier transform of this? 00:37:35.230 --> 00:37:41.380 align:middle line:84% So first, for this one, the Fourier transform is-- 00:37:41.380 --> 00:37:44.710 align:middle line:90% as we see, it's black. 00:37:44.710 --> 00:37:50.610 align:middle line:84% And then if you take that in 2D-- 00:37:50.610 --> 00:37:53.705 align:middle line:90% so how is the code? 00:37:53.705 --> 00:37:54.580 align:middle line:90% I'll give you a hint. 00:37:54.580 --> 00:37:57.280 align:middle line:84% If I just take a square aperture, a traditional one, 00:37:57.280 --> 00:37:59.960 align:middle line:84% and take a square transform, it will look-- 00:37:59.960 --> 00:38:02.815 align:middle line:84% the Fourier transform of this one looks something like this. 00:38:02.815 --> 00:38:05.680 align:middle line:90% 00:38:05.680 --> 00:38:06.280 align:middle line:90% [INAUDIBLE] 00:38:06.280 --> 00:38:08.600 align:middle line:84% So Fourier transform of this one-- 00:38:08.600 --> 00:38:10.090 align:middle line:84% if I take the cross-section here, 00:38:10.090 --> 00:38:12.286 align:middle line:90% it's going to look the same. 00:38:12.286 --> 00:38:14.880 align:middle line:84% Same thing here for a square aperture. 00:38:14.880 --> 00:38:21.695 align:middle line:84% And now you're saying for this crossword-puzzle-shaped item, 00:38:21.695 --> 00:38:23.947 align:middle line:90% should be easy. 00:38:23.947 --> 00:38:25.530 align:middle line:84% It's going to look just like this one. 00:38:25.530 --> 00:38:32.110 align:middle line:90% 00:38:32.110 --> 00:38:36.625 align:middle line:84% So a Fourier transform of 7 by 7 will have a peak in the middle. 00:38:36.625 --> 00:38:39.790 align:middle line:84% So the truly Fourier transform will have a peak in the middle. 00:38:39.790 --> 00:38:43.005 align:middle line:84% But the rest of the values will be constant. 00:38:43.005 --> 00:38:46.198 align:middle line:84% And that's the magic of a broadband code. 00:38:46.198 --> 00:38:47.740 align:middle line:84% So if we're placing a broadband code, 00:38:47.740 --> 00:38:52.330 align:middle line:84% certainly we have an opportunity to recover all the information. 00:38:52.330 --> 00:38:57.520 align:middle line:84% So it seems very, very long winded, right? 00:38:57.520 --> 00:39:01.360 align:middle line:84% If all I wanted to do was create a photo from which 00:39:01.360 --> 00:39:06.130 align:middle line:84% I can deblur to get sharp photo, why 00:39:06.130 --> 00:39:10.300 align:middle line:84% do I need to think about all this theory, right? 00:39:10.300 --> 00:39:14.213 align:middle line:84% And the reason is, when we think about point spread function, 00:39:14.213 --> 00:39:15.880 align:middle line:84% it's just traditional signal processing. 00:39:15.880 --> 00:39:18.730 align:middle line:84% It's a convolution and so on, and it's much easier 00:39:18.730 --> 00:39:22.930 align:middle line:84% to think about convolution and deconvolution in frequency 00:39:22.930 --> 00:39:25.480 align:middle line:90% domain than in primal domain. 00:39:25.480 --> 00:39:30.350 align:middle line:84% And in communication theory, everything is [INAUDIBLE].. 00:39:30.350 --> 00:39:33.130 align:middle line:84% We think about carrier frequencies of radio stations 00:39:33.130 --> 00:39:33.790 align:middle line:90% in frequencies. 00:39:33.790 --> 00:39:37.390 align:middle line:84% We say, my FM channel is at 99 megahertz, 100 megahertz, 00:39:37.390 --> 00:39:38.900 align:middle line:90% and so on. 00:39:38.900 --> 00:39:42.643 align:middle line:84% And we think about guard bands and audio bands 00:39:42.643 --> 00:39:44.560 align:middle line:84% and everything interested in frequency domain. 00:39:44.560 --> 00:39:46.330 align:middle line:84% And that's because it's signal processing. 00:39:46.330 --> 00:39:48.280 align:middle line:84% It's the same thing that's going on here. 00:39:48.280 --> 00:39:50.800 align:middle line:84% And convolution, deconvolution-- much easier 00:39:50.800 --> 00:39:52.630 align:middle line:90% to think in frequency domain. 00:39:52.630 --> 00:39:55.600 align:middle line:84% Although all the analysis in the frequency domain, at the end, 00:39:55.600 --> 00:39:57.340 align:middle line:90% the solution is very easy-- 00:39:57.340 --> 00:40:02.388 align:middle line:84% just flutter the shutter or just put a coded aperture. 00:40:02.388 --> 00:40:06.110 align:middle line:84% Extremely simple solution to achieve that. 00:40:06.110 --> 00:40:09.890 align:middle line:84% So those are all good things about coded aperture. 00:40:09.890 --> 00:40:12.625 align:middle line:84% What are some bad things about coded aperture? 00:40:12.625 --> 00:40:14.000 align:middle line:84% What are some disadvantages here? 00:40:14.000 --> 00:40:16.967 align:middle line:90% 00:40:16.967 --> 00:40:18.550 align:middle line:84% It's very similar to the [INAUDIBLE].. 00:40:18.550 --> 00:40:20.035 align:middle line:90% AUDIENCE: Half the light. 00:40:20.035 --> 00:40:21.640 align:middle line:84% RAMESH RASKAR: So half the light. 00:40:21.640 --> 00:40:22.150 align:middle line:90% Very good. 00:40:22.150 --> 00:40:25.253 align:middle line:84% And that's when you talk to people who build cameras, 00:40:25.253 --> 00:40:26.920 align:middle line:84% and you tell them, they say, no, no, no. 00:40:26.920 --> 00:40:28.130 align:middle line:90% That's not allowed. 00:40:28.130 --> 00:40:30.240 align:middle line:90% It doesn't cut the light. 00:40:30.240 --> 00:40:30.870 align:middle line:90% Yes. 00:40:30.870 --> 00:40:32.573 align:middle line:84% AUDIENCE: Are the bokehs kind of uppity? 00:40:32.573 --> 00:40:34.740 align:middle line:84% RAMESH RASKAR: The bokehs are-- it depends on your-- 00:40:34.740 --> 00:40:36.930 align:middle line:84% I mean, for your average consumer, 00:40:36.930 --> 00:40:38.370 align:middle line:84% I don't know whether this matters. 00:40:38.370 --> 00:40:39.078 align:middle line:90% But you're right. 00:40:39.078 --> 00:40:40.830 align:middle line:84% If you're looking at something that's-- 00:40:40.830 --> 00:40:42.690 align:middle line:84% we have bright lights in the scene. 00:40:42.690 --> 00:40:44.622 align:middle line:84% At a distance, take our false photo. 00:40:44.622 --> 00:40:45.830 align:middle line:90% They will all look like this. 00:40:45.830 --> 00:40:47.850 align:middle line:84% AUDIENCE: Or you could put hearts in it, or, like-- 00:40:47.850 --> 00:40:49.080 align:middle line:84% AUDIENCE: Right, yeah, I was thinking maybe-- 00:40:49.080 --> 00:40:50.830 align:middle line:84% AUDIENCE: I mean, that's totally possibly. 00:40:50.830 --> 00:40:52.500 align:middle line:90% [LAUGHTER] 00:40:52.500 --> 00:40:54.390 align:middle line:84% RAMESH RASKAR: So an interesting art problem 00:40:54.390 --> 00:40:55.560 align:middle line:90% is how do you create-- 00:40:55.560 --> 00:41:00.450 align:middle line:84% how do you create a mask that visually looks aesthetic but is 00:41:00.450 --> 00:41:01.830 align:middle line:90% mathematically also invertible. 00:41:01.830 --> 00:41:03.210 align:middle line:90% AUDIENCE: Yeah. 00:41:03.210 --> 00:41:06.570 align:middle line:84% RAMESH RASKAR: Are there disadvantages? 00:41:06.570 --> 00:41:08.340 align:middle line:90% Or challenges? 00:41:08.340 --> 00:41:10.200 align:middle line:90% Not really disadvantage. 00:41:10.200 --> 00:41:11.700 align:middle line:84% Remember, in the motion case, we had 00:41:11.700 --> 00:41:14.340 align:middle line:90% to know how much the motion is. 00:41:14.340 --> 00:41:15.690 align:middle line:90% What do we need to know here? 00:41:15.690 --> 00:41:17.470 align:middle line:84% AUDIENCE: We know how much the blur is. 00:41:17.470 --> 00:41:18.970 align:middle line:84% RAMESH RASKAR: How much the blur is. 00:41:18.970 --> 00:41:20.490 align:middle line:90% And what is that function of? 00:41:20.490 --> 00:41:22.590 align:middle line:84% If anything, plane of focus is sharp. 00:41:22.590 --> 00:41:24.930 align:middle line:84% When it's out of plane of focus, it's blurred. 00:41:24.930 --> 00:41:27.090 align:middle line:84% But the size of the blur is dependent on what? 00:41:27.090 --> 00:41:27.828 align:middle line:90% AUDIENCE: Belt. 00:41:27.828 --> 00:41:29.370 align:middle line:84% RAMESH RASKAR: The belt. But not just 00:41:29.370 --> 00:41:32.340 align:middle line:84% depth-- depth from the plane of focus, right? 00:41:32.340 --> 00:41:34.830 align:middle line:84% So that's an extra parameter you would estimate somehow. 00:41:34.830 --> 00:41:37.230 align:middle line:84% Maybe you can use a rangefinder or something 00:41:37.230 --> 00:41:39.570 align:middle line:90% like that, or just a software. 00:41:39.570 --> 00:41:42.150 align:middle line:84% There are methods you can employ. 00:41:42.150 --> 00:41:45.480 align:middle line:84% AUDIENCE: Don't you just try to assume something like we've 00:41:45.480 --> 00:41:46.867 align:middle line:90% got to see this contrast? 00:41:46.867 --> 00:41:47.700 align:middle line:90% RAMESH RASKAR: Yeah. 00:41:47.700 --> 00:41:48.325 align:middle line:90% AUDIENCE: Yeah. 00:41:48.325 --> 00:41:49.700 align:middle line:84% RAMESH RASKAR: You could do that. 00:41:49.700 --> 00:41:50.910 align:middle line:90% It doesn't work that well. 00:41:50.910 --> 00:41:51.690 align:middle line:90% but you're right. 00:41:51.690 --> 00:41:55.610 align:middle line:84% That would be another way, too, to try this. 00:41:55.610 --> 00:41:58.372 align:middle line:84% AUDIENCE: You can just maximize your hard edges in the image. 00:41:58.372 --> 00:41:59.330 align:middle line:90% RAMESH RASKAR: Exactly. 00:41:59.330 --> 00:42:01.070 align:middle line:84% That's what you would do, like, in a light field, when 00:42:01.070 --> 00:42:02.390 align:middle line:90% we did the refocusing. 00:42:02.390 --> 00:42:03.710 align:middle line:90% That's the trick we used. 00:42:03.710 --> 00:42:05.640 align:middle line:84% We said, OK, let me try to refocus. 00:42:05.640 --> 00:42:07.550 align:middle line:90% I don't care about the depth. 00:42:07.550 --> 00:42:10.460 align:middle line:84% When it comes into sharp focus, my edges, 00:42:10.460 --> 00:42:12.170 align:middle line:90% that must be the right depth. 00:42:12.170 --> 00:42:14.990 align:middle line:84% Unfortunately, it doesn't work out in this case. 00:42:14.990 --> 00:42:18.110 align:middle line:84% And we won't go into the detail, but the main reason 00:42:18.110 --> 00:42:21.050 align:middle line:84% is that, because it's coded aperture, no matter 00:42:21.050 --> 00:42:23.120 align:middle line:84% where you refocus, it still looks like it 00:42:23.120 --> 00:42:24.230 align:middle line:90% has very high frequencies. 00:42:24.230 --> 00:42:27.080 align:middle line:90% 00:42:27.080 --> 00:42:28.990 align:middle line:90% So that makes it challenging. 00:42:28.990 --> 00:42:29.490 align:middle line:90% Yes. 00:42:29.490 --> 00:42:31.450 align:middle line:84% AUDIENCE: How did you come up with the pattern? 00:42:31.450 --> 00:42:32.575 align:middle line:90% RAMESH RASKAR: Oh, exactly. 00:42:32.575 --> 00:42:35.200 align:middle line:84% So you need to find this 7-by-7 pattern or even 00:42:35.200 --> 00:42:37.240 align:middle line:84% the previous case, the 52 pattern. 00:42:37.240 --> 00:42:39.640 align:middle line:90% And you take a random sequence. 00:42:39.640 --> 00:42:41.522 align:middle line:84% Take a Fourier transform to see if it's flat. 00:42:41.522 --> 00:42:43.230 align:middle line:84% If it's not flat, you go to the next one. 00:42:43.230 --> 00:42:44.090 align:middle line:84% AUDIENCE: Oh, so this is brute force? 00:42:44.090 --> 00:42:46.020 align:middle line:84% There's not, like, a pretty mathematical formula for this? 00:42:46.020 --> 00:42:47.895 align:middle line:84% RAMESH RASKAR: So initial, that's what I did. 00:42:47.895 --> 00:42:49.660 align:middle line:84% I said, wow, it can't be that bad. 00:42:49.660 --> 00:42:50.530 align:middle line:90% 2 to the 50-- 00:42:50.530 --> 00:42:52.180 align:middle line:90% I mean, it's 52-element long. 00:42:52.180 --> 00:42:53.740 align:middle line:90% And I know some of them. 00:42:53.740 --> 00:42:56.888 align:middle line:84% I only want to take the ones in which about half of them are-- 00:42:56.888 --> 00:42:57.430 align:middle line:90% AUDIENCE: 1s. 00:42:57.430 --> 00:42:59.222 align:middle line:84% RAMESH RASKAR: --1s and half of them are 0. 00:42:59.222 --> 00:43:00.500 align:middle line:90% So it can't be that bad. 00:43:00.500 --> 00:43:02.510 align:middle line:90% So I wrote a MATLAB script. 00:43:02.510 --> 00:43:04.930 align:middle line:84% And I said, by the time I come tomorrow morning, 00:43:04.930 --> 00:43:06.370 align:middle line:90% I'll find a really good code. 00:43:06.370 --> 00:43:07.685 align:middle line:90% And I came back next morning. 00:43:07.685 --> 00:43:08.560 align:middle line:90% Nothing had happened. 00:43:08.560 --> 00:43:09.268 align:middle line:90% I waited all day. 00:43:09.268 --> 00:43:10.720 align:middle line:90% It was still running. 00:43:10.720 --> 00:43:12.230 align:middle line:90% And it never came out of that. 00:43:12.230 --> 00:43:15.760 align:middle line:84% So 2 the 52 is pretty challenging. 00:43:15.760 --> 00:43:16.390 align:middle line:90% AUDIENCE: Yes. 00:43:16.390 --> 00:43:16.970 align:middle line:90% Where's your [? canu ?] cluster? 00:43:16.970 --> 00:43:17.770 align:middle line:90% We need it. 00:43:17.770 --> 00:43:19.300 align:middle line:90% RAMESH RASKAR: Yeah, so-- sorry? 00:43:19.300 --> 00:43:20.530 align:middle line:84% AUDIENCE: Where's your [? canu ?] cluster? 00:43:20.530 --> 00:43:21.040 align:middle line:90% We need it. 00:43:21.040 --> 00:43:21.640 align:middle line:90% RAMESH RASKAR: Exactly. 00:43:21.640 --> 00:43:24.223 align:middle line:84% But even if you use a cluster, it's still a pretty big number. 00:43:24.223 --> 00:43:25.700 align:middle line:84% So you can do some approximation. 00:43:25.700 --> 00:43:29.320 align:middle line:84% So you can start with some code and do a gradient descent 00:43:29.320 --> 00:43:30.650 align:middle line:90% and so on. 00:43:30.650 --> 00:43:31.150 align:middle line:90% Yeah. 00:43:31.150 --> 00:43:33.160 align:middle line:84% AUDIENCE: Does the [? harder ?] [? mark ?] code or anything? 00:43:33.160 --> 00:43:34.448 align:middle line:90% Is that applicable here? 00:43:34.448 --> 00:43:35.240 align:middle line:90% RAMESH RASKAR: Mhm. 00:43:35.240 --> 00:43:39.850 align:middle line:84% So actually, after we did these two projects, 00:43:39.850 --> 00:43:42.100 align:middle line:84% I attended Professor Han's lecture 00:43:42.100 --> 00:43:45.340 align:middle line:84% on computational imaging, which I highly recommend, by the way. 00:43:45.340 --> 00:43:46.780 align:middle line:90% It's terrific. 00:43:46.780 --> 00:43:50.800 align:middle line:84% And there are all these theories about how 00:43:50.800 --> 00:43:53.660 align:middle line:84% to create different codes for different applications. 00:43:53.660 --> 00:43:56.980 align:middle line:84% So [? harder ?] [? mark ?] code, which we learned about a few 00:43:56.980 --> 00:44:00.040 align:middle line:84% weeks ago or so-called broadband codes, 00:44:00.040 --> 00:44:03.430 align:middle line:84% they all have polynomial solutions and this and that. 00:44:03.430 --> 00:44:05.230 align:middle line:84% There's no good solutions for 2D. 00:44:05.230 --> 00:44:09.040 align:middle line:84% But for 1D, there are some really good solutions 00:44:09.040 --> 00:44:11.120 align:middle line:90% to come up with that. 00:44:11.120 --> 00:44:14.020 align:middle line:84% And even for 2D, for certain dimensions, 00:44:14.020 --> 00:44:16.810 align:middle line:84% they call it one more 4 or three more 00:44:16.810 --> 00:44:19.870 align:middle line:84% 4 because prime numbers can be one more 4. 00:44:19.870 --> 00:44:25.540 align:middle line:84% Basically, when you divide by 4, the remainder can be 1 or 3. 00:44:25.540 --> 00:44:27.610 align:middle line:84% And there are certain sequences that 00:44:27.610 --> 00:44:29.440 align:middle line:84% are beautiful mathematical properties, 00:44:29.440 --> 00:44:32.680 align:middle line:84% of which sequences could have broadband properties 00:44:32.680 --> 00:44:34.750 align:middle line:90% and which may not. 00:44:34.750 --> 00:44:36.430 align:middle line:90% So it turns out you cannot-- 00:44:36.430 --> 00:44:38.870 align:middle line:84% there's a little bit of cheating going on here. 00:44:38.870 --> 00:44:42.310 align:middle line:84% So you cannot really use the broadband code here either 00:44:42.310 --> 00:44:45.160 align:middle line:84% to give you the best result. You can call them broadband 00:44:45.160 --> 00:44:46.750 align:middle line:84% because their behavior is broadband. 00:44:46.750 --> 00:44:51.730 align:middle line:84% But the traditional code's called MURA code, M-U-R-A, 00:44:51.730 --> 00:44:53.680 align:middle line:84% Multiple Uniform Redundant Array. 00:44:53.680 --> 00:44:57.940 align:middle line:84% They invented not very long ago, maybe 20, 30 years ago. 00:44:57.940 --> 00:45:05.680 align:middle line:84% And they used in CDMA and many other astronomical-imaging 00:45:05.680 --> 00:45:06.490 align:middle line:90% applications. 00:45:06.490 --> 00:45:09.850 align:middle line:84% And they have similar properties of being-- 00:45:09.850 --> 00:45:12.820 align:middle line:84% if you take a circle to transform, it's broadband. 00:45:12.820 --> 00:45:17.810 align:middle line:84% The problem is, in many of those example-- 00:45:17.810 --> 00:45:20.470 align:middle line:84% many of those applications, your convolution 00:45:20.470 --> 00:45:21.820 align:middle line:90% is actually circular. 00:45:21.820 --> 00:45:25.090 align:middle line:84% So you apply the filter, and then when you go off the edge, 00:45:25.090 --> 00:45:29.600 align:middle line:84% you apply the filter to the beginning of the signal. 00:45:29.600 --> 00:45:31.930 align:middle line:84% This particular filter is actually not circular, 00:45:31.930 --> 00:45:32.740 align:middle line:90% but it's linear. 00:45:32.740 --> 00:45:34.660 align:middle line:84% So when you apply the filter here, 00:45:34.660 --> 00:45:37.300 align:middle line:84% when you start applying the filter at the end of the image, 00:45:37.300 --> 00:45:39.970 align:middle line:84% you don't go back to the front of the image because, 00:45:39.970 --> 00:45:43.550 align:middle line:84% clearly, if I put an LED here, you get out of focus. 00:45:43.550 --> 00:45:46.408 align:middle line:84% If I put an LED here, you'll only get half of that. 00:45:46.408 --> 00:45:47.950 align:middle line:84% The rest of the half is just blocked. 00:45:47.950 --> 00:45:50.450 align:middle line:84% It's not going to magically appear over here. 00:45:50.450 --> 00:45:52.810 align:middle line:84% So that's the difference between linear convolution 00:45:52.810 --> 00:45:54.280 align:middle line:90% and circular convolution. 00:45:54.280 --> 00:45:56.920 align:middle line:84% It turns out, for circular convolution, 00:45:56.920 --> 00:46:00.760 align:middle line:84% the match is very clean and beautiful and smoother course 00:46:00.760 --> 00:46:01.390 align:middle line:90% work. 00:46:01.390 --> 00:46:04.810 align:middle line:84% Or for linear convolution, there is no good mechanism. 00:46:04.810 --> 00:46:10.030 align:middle line:84% So we came up with our own code called RAT code, R-A-T, 00:46:10.030 --> 00:46:13.180 align:middle line:90% which is after three quarters. 00:46:13.180 --> 00:46:14.720 align:middle line:90% Oscar [INAUDIBLE]. 00:46:14.720 --> 00:46:17.320 align:middle line:84% AUDIENCE: So how did you find that code? 00:46:17.320 --> 00:46:18.755 align:middle line:84% RAMESH RASKAR: By doing research. 00:46:18.755 --> 00:46:20.005 align:middle line:90% AUDIENCE: Just doing research? 00:46:20.005 --> 00:46:20.290 align:middle line:90% RAMESH RASKAR: Yeah. 00:46:20.290 --> 00:46:21.370 align:middle line:90% AUDIENCE: OK. 00:46:21.370 --> 00:46:23.415 align:middle line:84% RAMESH RASKAR: But it's not a brute-force search. 00:46:23.415 --> 00:46:24.040 align:middle line:90% AUDIENCE: Yeah. 00:46:24.040 --> 00:46:25.300 align:middle line:90% It was an intelligent. 00:46:25.300 --> 00:46:29.890 align:middle line:84% AUDIENCE: And if you included enough padding there, 00:46:29.890 --> 00:46:32.425 align:middle line:84% wouldn't you be able to use circular convolution? 00:46:32.425 --> 00:46:34.550 align:middle line:84% RAMESH RASKAR: Yeah, I mean, circular convolution-- 00:46:34.550 --> 00:46:38.140 align:middle line:84% I mean the linear convolution is basically circular convolution 00:46:38.140 --> 00:46:39.415 align:middle line:90% with a lot of padding of 0s. 00:46:39.415 --> 00:46:42.040 align:middle line:84% AUDIENCE: Yeah, because you said then the math would be easier, 00:46:42.040 --> 00:46:42.340 align:middle line:90% right? 00:46:42.340 --> 00:46:43.965 align:middle line:84% RAMESH RASKAR: But then it's too large. 00:46:43.965 --> 00:46:49.990 align:middle line:84% I mean, finding a code that's 7 long or maybe 30 long is OK. 00:46:49.990 --> 00:46:52.628 align:middle line:84% Finding a code that's 1,000 long is nearly impossible. 00:46:52.628 --> 00:46:54.670 align:middle line:84% AUDIENCE: So the difference between MULA and that 00:46:54.670 --> 00:46:55.870 align:middle line:90% is only on the edges? 00:46:55.870 --> 00:46:57.250 align:middle line:90% Or is it all over the picture? 00:46:57.250 --> 00:46:58.150 align:middle line:84% RAMESH RASKAR: It's only the fact 00:46:58.150 --> 00:46:59.940 align:middle line:84% that one is linear convolution and one 00:46:59.940 --> 00:47:00.940 align:middle line:90% is circular convolution. 00:47:00.940 --> 00:47:01.757 align:middle line:90% AUDIENCE: OK. 00:47:01.757 --> 00:47:04.090 align:middle line:84% AUDIENCE: Yeah, and another thing is it's pretty amazing 00:47:04.090 --> 00:47:08.680 align:middle line:84% that [INAUDIBLE] because if you start just having very simple 00:47:08.680 --> 00:47:11.020 align:middle line:84% patterns on a square-- like, say, 00:47:11.020 --> 00:47:13.510 align:middle line:84% if you just draw this square and sat down this square, 00:47:13.510 --> 00:47:16.113 align:middle line:84% you get the free entrance form, and you have all-- 00:47:16.113 --> 00:47:17.780 align:middle line:84% RAMESH RASKAR: Yeah, all over the place. 00:47:17.780 --> 00:47:21.670 align:middle line:84% So, yeah, so it seems like can just choose a random sequence 00:47:21.670 --> 00:47:23.170 align:middle line:90% and get a similar property. 00:47:23.170 --> 00:47:24.430 align:middle line:90% But actually, it doesn't work. 00:47:24.430 --> 00:47:28.070 align:middle line:84% The chances of a random sequence doing the right thing for you 00:47:28.070 --> 00:47:29.616 align:middle line:90% is very, very low. 00:47:29.616 --> 00:47:31.480 align:middle line:84% AUDIENCE: Instead of [INAUDIBLE].. 00:47:31.480 --> 00:47:33.360 align:middle line:90% [LAUGHTER] 00:47:33.360 --> 00:47:36.780 align:middle line:90% 00:47:36.780 --> 00:47:38.910 align:middle line:84% AUDIENCE: Are astronomy people are already using-- 00:47:38.910 --> 00:47:39.952 align:middle line:90% RAMESH RASKAR: Mhm, yeah. 00:47:39.952 --> 00:47:42.610 align:middle line:84% AUDIENCE: Or they were using this for [INAUDIBLE]?? 00:47:42.610 --> 00:47:45.960 align:middle line:84% RAMESH RASKAR: So in astronomy, you have circular convolution 00:47:45.960 --> 00:47:48.780 align:middle line:84% because they use either two mirror tiles 00:47:48.780 --> 00:47:52.150 align:middle line:84% and one sensor or one mirror tile and two sensors. 00:47:52.150 --> 00:47:55.270 align:middle line:84% So the whole circular convolution. 00:47:55.270 --> 00:47:56.670 align:middle line:90% So all right. 00:47:56.670 --> 00:47:59.440 align:middle line:84% AUDIENCE: If you're tired of [INAUDIBLE] astronomically 00:47:59.440 --> 00:48:01.165 align:middle line:90% coded imagery. 00:48:01.165 --> 00:48:02.290 align:middle line:90% RAMESH RASKAR: Repeat that. 00:48:02.290 --> 00:48:05.650 align:middle line:84% AUDIENCE: If you're tiling the mask at aperture, 00:48:05.650 --> 00:48:07.510 align:middle line:84% but you are using single-tiled aperture-- 00:48:07.510 --> 00:48:08.140 align:middle line:90% RAMESH RASKAR: Right. 00:48:08.140 --> 00:48:09.700 align:middle line:84% AUDIENCE: So if you're tiling that up-- 00:48:09.700 --> 00:48:10.630 align:middle line:84% RAMESH RASKAR: If you tile aperture, 00:48:10.630 --> 00:48:12.940 align:middle line:84% you'll get really horrible frequency response, 00:48:12.940 --> 00:48:16.480 align:middle line:84% unfortunately, because if you put two tiles, that means 00:48:16.480 --> 00:48:17.740 align:middle line:90% certain frequencies are lost. 00:48:17.740 --> 00:48:21.460 align:middle line:90% 00:48:21.460 --> 00:48:23.290 align:middle line:84% AUDIENCE: [INAUDIBLE] impressive. 00:48:23.290 --> 00:48:26.620 align:middle line:84% It's saying that, if I understand this right, 00:48:26.620 --> 00:48:29.680 align:middle line:84% basically, by taking the DC coefficient, 00:48:29.680 --> 00:48:32.570 align:middle line:84% you're reconstructing almost everything. 00:48:32.570 --> 00:48:33.070 align:middle line:90% Is that-- 00:48:33.070 --> 00:48:34.030 align:middle line:84% RAMESH RASKAR: No, no, no, not DC 00:48:34.030 --> 00:48:36.700 align:middle line:84% coefficient because if you look here, all the high spatial-- 00:48:36.700 --> 00:48:39.130 align:middle line:84% I mean, the whole image is not one value. 00:48:39.130 --> 00:48:40.720 align:middle line:84% AUDIENCE: Yeah, but look at that. 00:48:40.720 --> 00:48:43.145 align:middle line:90% That's the spectrum of your-- 00:48:43.145 --> 00:48:44.020 align:middle line:90% RAMESH RASKAR: Right. 00:48:44.020 --> 00:48:46.453 align:middle line:84% No, but there is a non-zero value at other frequencies. 00:48:46.453 --> 00:48:47.620 align:middle line:90% AUDIENCE: Yeah, yeah, a few. 00:48:47.620 --> 00:48:48.730 align:middle line:90% But-- 00:48:48.730 --> 00:48:49.730 align:middle line:84% RAMESH RASKAR: No, no, that's very important. 00:48:49.730 --> 00:48:50.960 align:middle line:84% AUDIENCE: Yeah, but by taking that, 00:48:50.960 --> 00:48:52.630 align:middle line:84% you could get a very good approximation. 00:48:52.630 --> 00:48:53.463 align:middle line:90% RAMESH RASKAR: Yeah. 00:48:53.463 --> 00:48:59.440 align:middle line:84% But if-- to a naive consumer, this photo-- 00:48:59.440 --> 00:49:00.520 align:middle line:90% so look at this part, OK? 00:49:00.520 --> 00:49:04.480 align:middle line:84% This photo and this photo looks almost the same, right? 00:49:04.480 --> 00:49:08.140 align:middle line:84% And remember, in this photo, many of those frequencies 00:49:08.140 --> 00:49:08.740 align:middle line:90% are lost. 00:49:08.740 --> 00:49:11.710 align:middle line:90% 00:49:11.710 --> 00:49:14.410 align:middle line:84% And in this photo, those frequencies 00:49:14.410 --> 00:49:18.910 align:middle line:84% are not lost because all the frequencies are preserved. 00:49:18.910 --> 00:49:20.530 align:middle line:84% But that's because our eyes are not 00:49:20.530 --> 00:49:26.050 align:middle line:84% very good at thinking about what the original image could be, 00:49:26.050 --> 00:49:28.150 align:middle line:84% given either this one or the previous one. 00:49:28.150 --> 00:49:31.240 align:middle line:84% So given this, I can challenge you 00:49:31.240 --> 00:49:33.430 align:middle line:84% that you're not able to predict that it 00:49:33.430 --> 00:49:37.850 align:middle line:90% has all this structure, right? 00:49:37.850 --> 00:49:42.450 align:middle line:84% From here, you cannot predict that you have the structure. 00:49:42.450 --> 00:49:45.100 align:middle line:84% AUDIENCE: So how would you describe the mask as? 00:49:45.100 --> 00:49:47.870 align:middle line:84% Basically, you spread the energy in [INAUDIBLE] 00:49:47.870 --> 00:49:53.920 align:middle line:84% set of over many frequencies but very small coefficients. 00:49:53.920 --> 00:49:54.420 align:middle line:90% Is that-- 00:49:54.420 --> 00:49:55.378 align:middle line:90% RAMESH RASKAR: Exactly. 00:49:55.378 --> 00:49:59.960 align:middle line:84% It's about-- depending on the code, it's about 1/10 or 1/20 00:49:59.960 --> 00:50:03.650 align:middle line:84% of the original power of that frequency. 00:50:03.650 --> 00:50:05.720 align:middle line:84% So you get significant attenuation. 00:50:05.720 --> 00:50:07.460 align:middle line:90% So the results are not perfect. 00:50:07.460 --> 00:50:11.910 align:middle line:84% If you look here, right, it's not it's not perfect results, 00:50:11.910 --> 00:50:15.710 align:middle line:90% whether it's here or here. 00:50:15.710 --> 00:50:17.540 align:middle line:90% Look at this. 00:50:17.540 --> 00:50:19.370 align:middle line:84% I wouldn't call it photography quality yet. 00:50:19.370 --> 00:50:19.910 align:middle line:90% AUDIENCE: Yeah, no. 00:50:19.910 --> 00:50:21.980 align:middle line:84% RAMESH RASKAR: But if you apply very simple-- 00:50:21.980 --> 00:50:23.150 align:middle line:90% but it is a raw resource. 00:50:23.150 --> 00:50:25.500 align:middle line:84% There is no medium filtering or smoothing or anything. 00:50:25.500 --> 00:50:31.970 align:middle line:84% It's just pure x equals b, x equals a backslash b. 00:50:31.970 --> 00:50:34.570 align:middle line:84% AUDIENCE: Just the fact that the mask, I guess, 00:50:34.570 --> 00:50:35.600 align:middle line:90% gives you balance. 00:50:35.600 --> 00:50:37.790 align:middle line:90% RAMESH RASKAR: Yeah, it's fun. 00:50:37.790 --> 00:50:39.500 align:middle line:84% What's amazing about coded imaging 00:50:39.500 --> 00:50:43.430 align:middle line:84% is that the math is elegant and beautiful and sometimes 00:50:43.430 --> 00:50:46.100 align:middle line:84% complicated, but the implementation is very easy. 00:50:46.100 --> 00:50:48.200 align:middle line:84% At the end, all I had to do is put this code 00:50:48.200 --> 00:50:51.530 align:middle line:84% or shutter it, and very easy to explain. 00:50:51.530 --> 00:50:53.870 align:middle line:84% My previous boss is to say, the best ideas 00:50:53.870 --> 00:50:55.730 align:middle line:84% are the ones that are easy to explain 00:50:55.730 --> 00:50:58.970 align:middle line:90% but difficult to conceive. 00:50:58.970 --> 00:51:01.490 align:middle line:90% All right, so let's move on. 00:51:01.490 --> 00:51:03.260 align:middle line:90% OK, let me finish this one. 00:51:03.260 --> 00:51:07.460 align:middle line:90% So there's just one way of-- 00:51:07.460 --> 00:51:09.650 align:middle line:84% we only saw two ways of engineering 00:51:09.650 --> 00:51:13.970 align:middle line:84% the point spread function, one in motion and one in focus, 00:51:13.970 --> 00:51:14.630 align:middle line:90% right? 00:51:14.630 --> 00:51:16.070 align:middle line:90% But there are many others. 00:51:16.070 --> 00:51:19.250 align:middle line:84% We saw some of them over the course of the semester, 00:51:19.250 --> 00:51:26.570 align:middle line:84% where you can put, for example, a special filter in the lens 00:51:26.570 --> 00:51:30.565 align:middle line:84% so that you get blur that's independent of that. 00:51:30.565 --> 00:51:32.690 align:middle line:84% AUDIENCE: Ramesh, let me ask you one more question. 00:51:32.690 --> 00:51:33.898 align:middle line:90% RAMESH RASKAR: Yes, go ahead. 00:51:33.898 --> 00:51:35.720 align:middle line:84% AUDIENCE: You had this binary mask, right? 00:51:35.720 --> 00:51:36.200 align:middle line:90% RAMESH RASKAR: Mhm. 00:51:36.200 --> 00:51:37.970 align:middle line:84% AUDIENCE: What if the mask was not quite? 00:51:37.970 --> 00:51:40.490 align:middle line:84% If you have some information by the board so that 00:51:40.490 --> 00:51:42.885 align:middle line:84% you could set up approximate [INAUDIBLE].. 00:51:42.885 --> 00:51:43.760 align:middle line:90% RAMESH RASKAR: Right. 00:51:43.760 --> 00:51:45.090 align:middle line:84% AUDIENCE: So what would you have? 00:51:45.090 --> 00:51:47.007 align:middle line:84% RAMESH RASKAR: So that's a very good question. 00:51:47.007 --> 00:51:48.274 align:middle line:90% So-- let's see. 00:51:48.274 --> 00:51:51.540 align:middle line:90% Let me get this out first. 00:51:51.540 --> 00:51:53.435 align:middle line:90% So if the function was-- 00:51:53.435 --> 00:52:05.430 align:middle line:90% 00:52:05.430 --> 00:52:07.800 align:middle line:90% if the function was continuous-- 00:52:07.800 --> 00:52:09.752 align:middle line:84% so in case of flutter shutter, we 00:52:09.752 --> 00:52:10.960 align:middle line:90% didn't have much of a choice. 00:52:10.960 --> 00:52:12.493 align:middle line:84% It's either opaque or transparent. 00:52:12.493 --> 00:52:13.410 align:middle line:90% It's one or the other. 00:52:13.410 --> 00:52:14.285 align:middle line:90% AUDIENCE: Yeah, yeah. 00:52:14.285 --> 00:52:17.370 align:middle line:84% RAMESH RASKAR: But in case of aperture, yes. 00:52:17.370 --> 00:52:19.320 align:middle line:84% It doesn't have to be opaque or transparent. 00:52:19.320 --> 00:52:22.800 align:middle line:90% It could be a continuous value. 00:52:22.800 --> 00:52:27.390 align:middle line:84% And initially, actually, I and my co-author, Amit Agrawal-- 00:52:27.390 --> 00:52:30.690 align:middle line:84% very smart guy-- we always had these arguments about maybe 00:52:30.690 --> 00:52:31.590 align:middle line:90% continuous is better. 00:52:31.590 --> 00:52:32.880 align:middle line:90% Maybe binary is better. 00:52:32.880 --> 00:52:36.270 align:middle line:84% And he continued to believe that continuous is better. 00:52:36.270 --> 00:52:38.520 align:middle line:84% But it turns out-- and we still don't agree with this, 00:52:38.520 --> 00:52:39.960 align:middle line:90% by the way. 00:52:39.960 --> 00:52:43.200 align:middle line:84% And nobody has written this down. 00:52:43.200 --> 00:52:46.320 align:middle line:84% It turns out that, for any continuous code, 00:52:46.320 --> 00:52:48.870 align:middle line:84% there is a corresponding binary code that will 00:52:48.870 --> 00:52:52.230 align:middle line:90% do an equally good job, so far. 00:52:52.230 --> 00:52:55.003 align:middle line:84% And that's because in a binary code, 00:52:55.003 --> 00:52:56.670 align:middle line:84% you get to play with the phase function. 00:52:56.670 --> 00:52:57.810 align:middle line:90% I won't go to the detail. 00:52:57.810 --> 00:53:00.120 align:middle line:84% But because here, we are only showing you 00:53:00.120 --> 00:53:03.660 align:middle line:84% the amplitude of the Fourier transform but not the face. 00:53:03.660 --> 00:53:06.340 align:middle line:84% So you get that extra degree of freedom to play with. 00:53:06.340 --> 00:53:07.860 align:middle line:84% So if you play with the right phase, 00:53:07.860 --> 00:53:10.570 align:middle line:84% then it turns out you can always have a binary function. 00:53:10.570 --> 00:53:11.370 align:middle line:90% Mike? 00:53:11.370 --> 00:53:14.097 align:middle line:84% AUDIENCE: Has anyone tried to combine the coded aperture 00:53:14.097 --> 00:53:15.990 align:middle line:90% and the coded [INAUDIBLE]? 00:53:15.990 --> 00:53:18.820 align:middle line:84% RAMESH RASKAR: That's a great idea. 00:53:18.820 --> 00:53:23.750 align:middle line:84% People talk about it, but nobody has done it. 00:53:23.750 --> 00:53:25.000 align:middle line:90% It's just one of those things. 00:53:25.000 --> 00:53:28.150 align:middle line:90% 00:53:28.150 --> 00:53:29.650 align:middle line:90% It's just one of those things. 00:53:29.650 --> 00:53:34.348 align:middle line:84% It's like we are sick of it, so we don't want to do it. 00:53:34.348 --> 00:53:36.520 align:middle line:90% But I think it's worth trying. 00:53:36.520 --> 00:53:39.680 align:middle line:84% And because those are orthogonal motion blur. 00:53:39.680 --> 00:53:43.330 align:middle line:84% So here's a great thought experiment. 00:53:43.330 --> 00:53:46.900 align:middle line:84% So Mike's question was, there could 00:53:46.900 --> 00:53:52.330 align:middle line:84% be something that's moving, so it's motion blurred, but it's 00:53:52.330 --> 00:53:54.770 align:middle line:90% also out of focus. 00:53:54.770 --> 00:53:56.600 align:middle line:90% OK? 00:53:56.600 --> 00:54:00.887 align:middle line:84% Can you use both at the same time and record? 00:54:00.887 --> 00:54:01.470 align:middle line:90% AUDIENCE: Yes. 00:54:01.470 --> 00:54:03.250 align:middle line:84% AUDIENCE: What about the light width? 00:54:03.250 --> 00:54:04.310 align:middle line:84% RAMESH RASKAR: Yeah, it's one fourth of the light, 00:54:04.310 --> 00:54:05.602 align:middle line:90% but let's not worry about that. 00:54:05.602 --> 00:54:10.560 align:middle line:90% 00:54:10.560 --> 00:54:11.130 align:middle line:90% OK. 00:54:11.130 --> 00:54:11.880 align:middle line:90% [LAUGHTER] 00:54:11.880 --> 00:54:12.916 align:middle line:90% Explain. 00:54:12.916 --> 00:54:15.960 align:middle line:84% AUDIENCE: There are orthogonal technologies, basically. 00:54:15.960 --> 00:54:17.370 align:middle line:90% RAMESH RASKAR: Exactly. 00:54:17.370 --> 00:54:19.590 align:middle line:84% So it's amazing because motion is time, 00:54:19.590 --> 00:54:21.330 align:middle line:90% and the focus is space. 00:54:21.330 --> 00:54:23.600 align:middle line:90% They're completely orthogonal. 00:54:23.600 --> 00:54:24.600 align:middle line:90% So you can play with it. 00:54:24.600 --> 00:54:26.580 align:middle line:90% It's very interesting. 00:54:26.580 --> 00:54:28.230 align:middle line:84% AUDIENCE: But still, motion is being 00:54:28.230 --> 00:54:29.820 align:middle line:90% represented by space on the-- 00:54:29.820 --> 00:54:30.560 align:middle line:84% RAMESH RASKAR: Yeah, eventually, it's 00:54:30.560 --> 00:54:31.810 align:middle line:90% going to have a 2D projection. 00:54:31.810 --> 00:54:32.540 align:middle line:90% AUDIENCE: Yeah. 00:54:32.540 --> 00:54:34.392 align:middle line:84% RAMESH RASKAR: So that's very interesting. 00:54:34.392 --> 00:54:37.170 align:middle line:90% 00:54:37.170 --> 00:54:37.740 align:middle line:90% All right? 00:54:37.740 --> 00:54:44.310 align:middle line:84% So the point spread function, although I and my team 00:54:44.310 --> 00:54:47.310 align:middle line:84% were the first one to do that in a graphics vision domain, 00:54:47.310 --> 00:54:51.570 align:middle line:84% people have been trying to do that since mid '90s in imaging. 00:54:51.570 --> 00:54:54.900 align:middle line:84% And there was a very classic paper by Cathey and Dowski 00:54:54.900 --> 00:54:56.885 align:middle line:84% and others for so-called wavefront coding. 00:54:56.885 --> 00:54:58.260 align:middle line:84% And a lot of it is actually being 00:54:58.260 --> 00:55:00.450 align:middle line:90% used in cell phone cameras. 00:55:00.450 --> 00:55:02.100 align:middle line:84% And what they do is they put this face 00:55:02.100 --> 00:55:04.830 align:middle line:90% mask between the object-- 00:55:04.830 --> 00:55:06.992 align:middle line:90% near the lens so that-- 00:55:06.992 --> 00:55:08.950 align:middle line:84% and we saw this in the beginning of the class-- 00:55:08.950 --> 00:55:13.590 align:middle line:84% so that the image does not come into sharp focus ever. 00:55:13.590 --> 00:55:17.730 align:middle line:84% Instead of that, it's like a set of straws. 00:55:17.730 --> 00:55:20.580 align:middle line:84% Imagine these are all straws that are coming in. 00:55:20.580 --> 00:55:23.320 align:middle line:90% And you just twist them. 00:55:23.320 --> 00:55:25.298 align:middle line:84% So the top one kind of goes at the top-- 00:55:25.298 --> 00:55:26.340 align:middle line:90% I'm sorry, at the bottom. 00:55:26.340 --> 00:55:28.000 align:middle line:90% The bottom one goes at the top. 00:55:28.000 --> 00:55:31.800 align:middle line:84% And when you think about the cross-section of all 00:55:31.800 --> 00:55:33.990 align:middle line:84% the straws, it's kind of cylindrical, 00:55:33.990 --> 00:55:35.830 align:middle line:90% when they all come together. 00:55:35.830 --> 00:55:38.660 align:middle line:84% OK, I'm going to take all these straws, or maybe strings, 00:55:38.660 --> 00:55:39.910 align:middle line:90% if you want to think about it. 00:55:39.910 --> 00:55:43.590 align:middle line:84% And I'm going to twist them so that they remain cylindrical. 00:55:43.590 --> 00:55:47.130 align:middle line:84% So if I put my sensor here, if the image is out of focus 00:55:47.130 --> 00:55:49.920 align:middle line:84% by this width, if I put a sensor here, 00:55:49.920 --> 00:55:52.590 align:middle line:84% it's still out of focus but by the same width. 00:55:52.590 --> 00:55:55.020 align:middle line:84% So no matter where you are, the image 00:55:55.020 --> 00:55:59.750 align:middle line:84% is out of focus but by the same amount. 00:55:59.750 --> 00:56:02.145 align:middle line:84% And you say, well, what's good about that? 00:56:02.145 --> 00:56:03.530 align:middle line:90% It's always out of focus. 00:56:03.530 --> 00:56:06.180 align:middle line:84% But turns out, the wavefront coding, as they call it, 00:56:06.180 --> 00:56:09.170 align:middle line:84% but you can think of this now we know what light field. 00:56:09.170 --> 00:56:14.300 align:middle line:84% So this just a unique light field of the scene. 00:56:14.300 --> 00:56:17.420 align:middle line:84% It turns out that from that, you can recover images. 00:56:17.420 --> 00:56:20.150 align:middle line:90% 00:56:20.150 --> 00:56:22.160 align:middle line:90% Like, so this is open aperture. 00:56:22.160 --> 00:56:29.063 align:middle line:90% 00:56:29.063 --> 00:56:30.480 align:middle line:84% I'm sorry, I don't have a picture. 00:56:30.480 --> 00:56:32.800 align:middle line:84% But we discussed it in the class, 00:56:32.800 --> 00:56:34.050 align:middle line:90% so I hope you remember that. 00:56:34.050 --> 00:56:37.480 align:middle line:90% 00:56:37.480 --> 00:56:38.710 align:middle line:90% I missed that picture. 00:56:38.710 --> 00:56:42.280 align:middle line:84% We saw this right in the very first class, by the way. 00:56:42.280 --> 00:56:43.930 align:middle line:84% And the benefit of that, it turns out, 00:56:43.930 --> 00:56:47.530 align:middle line:84% is that it preserves the spatial frequencies, 00:56:47.530 --> 00:56:51.430 align:middle line:84% and it has the benefit that, no matter which steps you are at, 00:56:51.430 --> 00:56:53.950 align:middle line:90% you have the same defocus blur. 00:56:53.950 --> 00:56:56.020 align:middle line:84% So the disadvantage of coded aperture 00:56:56.020 --> 00:56:58.180 align:middle line:84% was that you need to know what the depth was 00:56:58.180 --> 00:56:59.950 align:middle line:90% to be able to deblur. 00:56:59.950 --> 00:57:02.320 align:middle line:84% But now, because it's independent of depth, 00:57:02.320 --> 00:57:05.350 align:middle line:84% you can just apply the same deconvolution 00:57:05.350 --> 00:57:07.180 align:middle line:90% and get back a sharper image. 00:57:07.180 --> 00:57:09.530 align:middle line:84% So whether if I hold myself on camera, 00:57:09.530 --> 00:57:12.350 align:middle line:84% whether I'm here or here or at infinity, 00:57:12.350 --> 00:57:14.770 align:middle line:90% I get the same amount of blur. 00:57:14.770 --> 00:57:17.290 align:middle line:90% Same point spread function. 00:57:17.290 --> 00:57:20.710 align:middle line:84% And from that, I can deconvolute and get 00:57:20.710 --> 00:57:24.220 align:middle line:84% an extended depth of field that goes from very 00:57:24.220 --> 00:57:28.080 align:middle line:90% close to the lens to infinity. 00:57:28.080 --> 00:57:34.280 align:middle line:84% So OmniVision, which bought this company, 00:57:34.280 --> 00:57:39.050 align:middle line:84% cerium optics, which is named after Cathey, Dowski, 00:57:39.050 --> 00:57:41.465 align:middle line:90% and somebody-- 00:57:41.465 --> 00:57:44.210 align:middle line:84% those are the two professors at Colorado. 00:57:44.210 --> 00:57:47.030 align:middle line:90% And the last one, I forget. 00:57:47.030 --> 00:57:49.730 align:middle line:84% That was just bought by OmniVision, 00:57:49.730 --> 00:57:52.345 align:middle line:90% which is a big cell phone-- 00:57:52.345 --> 00:57:53.960 align:middle line:90% I mean, big imaging company. 00:57:53.960 --> 00:57:55.880 align:middle line:84% Most of the business is cell phones. 00:57:55.880 --> 00:57:58.340 align:middle line:84% And they acquired the company and immediately 00:57:58.340 --> 00:58:01.955 align:middle line:84% laid off all the smart people who invented this. 00:58:01.955 --> 00:58:05.270 align:middle line:84% It's very sad Because that part is done. 00:58:05.270 --> 00:58:09.710 align:middle line:84% So they just wanted the technology. 00:58:09.710 --> 00:58:11.030 align:middle line:90% And it's in a lot of cameras. 00:58:11.030 --> 00:58:13.250 align:middle line:84% There's another company called Tessera, which 00:58:13.250 --> 00:58:17.430 align:middle line:90% has a very similar solution. 00:58:17.430 --> 00:58:20.160 align:middle line:84% But what they do is-- this one, basically what it does-- 00:58:20.160 --> 00:58:22.650 align:middle line:84% and we discussed this, I think, in the beginning, 00:58:22.650 --> 00:58:23.630 align:middle line:90% the wavefront coding-- 00:58:23.630 --> 00:58:28.976 align:middle line:84% is they are simply placing an addition here 00:58:28.976 --> 00:58:33.080 align:middle line:84% so that this part of the lens will focus on an image here. 00:58:33.080 --> 00:58:35.780 align:middle line:84% This part of the lens will focus on this one. 00:58:35.780 --> 00:58:37.520 align:middle line:90% This one focuses here. 00:58:37.520 --> 00:58:41.290 align:middle line:90% 00:58:41.290 --> 00:58:43.630 align:middle line:84% The top of the lens has a short focus lens. 00:58:43.630 --> 00:58:44.860 align:middle line:90% It focuses here. 00:58:44.860 --> 00:58:47.770 align:middle line:90% The second one focuses here. 00:58:47.770 --> 00:58:49.720 align:middle line:90% Third one focuses here. 00:58:49.720 --> 00:58:51.293 align:middle line:90% Fourth one focuses here. 00:58:51.293 --> 00:58:53.605 align:middle line:90% Fifth one focuses here. 00:58:53.605 --> 00:58:54.420 align:middle line:90% Here. 00:58:54.420 --> 00:58:56.220 align:middle line:90% And here. 00:58:56.220 --> 00:58:59.010 align:middle line:84% OK, so if you can imagine the main lens 00:58:59.010 --> 00:59:00.330 align:middle line:90% has a certain focal length. 00:59:00.330 --> 00:59:01.997 align:middle line:84% And we're just going to add a little bit 00:59:01.997 --> 00:59:05.550 align:middle line:84% of additional focal length, which is-- 00:59:05.550 --> 00:59:09.520 align:middle line:84% that's why you have focal length F1, F2, F10. 00:59:09.520 --> 00:59:11.110 align:middle line:90% And then [INAUDIBLE]. 00:59:11.110 --> 00:59:14.020 align:middle line:84% And this is the twist that I was talking about. 00:59:14.020 --> 00:59:15.406 align:middle line:90% This was [INAUDIBLE]. 00:59:15.406 --> 00:59:18.790 align:middle line:90% 00:59:18.790 --> 00:59:23.710 align:middle line:84% But within this region, the thickness will be a bonus. 00:59:23.710 --> 00:59:27.190 align:middle line:84% So you can either think of it as adding small matchsticks 00:59:27.190 --> 00:59:29.597 align:middle line:90% on top of the main lens-- 00:59:29.597 --> 00:59:31.180 align:middle line:84% or the way they do it is they actually 00:59:31.180 --> 00:59:37.860 align:middle line:84% put one single sheet that looks like that, an additional layer 00:59:37.860 --> 00:59:40.180 align:middle line:90% of support, a face mask. 00:59:40.180 --> 00:59:44.140 align:middle line:84% And a face mask basically means you are changing 00:59:44.140 --> 00:59:45.490 align:middle line:90% the face of incoming light. 00:59:45.490 --> 00:59:48.220 align:middle line:84% And, as you know, if you have a piece of glass, 00:59:48.220 --> 00:59:50.830 align:middle line:84% and light is going through, it's going 00:59:50.830 --> 00:59:54.350 align:middle line:84% to slow down here and then again [INAUDIBLE].. 00:59:54.350 --> 00:59:57.372 align:middle line:84% That means you basically slowed down the light. 00:59:57.372 --> 00:59:59.320 align:middle line:84% And that's where the glass [INAUDIBLE].. 00:59:59.320 --> 01:00:03.710 align:middle line:84% If you have it at the top of the lens, like those two, 01:00:03.710 --> 01:00:05.690 align:middle line:90% it doesn't slow down that much. 01:00:05.690 --> 01:00:10.040 align:middle line:84% If you go to the middle of it, it slows down forever. 01:00:10.040 --> 01:00:12.710 align:middle line:84% That's why, as we learned about at the beginning, 01:00:12.710 --> 01:00:15.320 align:middle line:84% if you have something very far away, this 01:00:15.320 --> 01:00:17.010 align:middle line:90% slows down a little bit. 01:00:17.010 --> 01:00:19.700 align:middle line:90% So those go over here. 01:00:19.700 --> 01:00:21.080 align:middle line:90% This goes over here. 01:00:21.080 --> 01:00:25.340 align:middle line:84% And everything just works out with operations. 01:00:25.340 --> 01:00:28.352 align:middle line:84% But [INAUDIBLE] this extra piece of glass, you're saying, 01:00:28.352 --> 01:00:29.810 align:middle line:84% I'm going to speed up and slow down 01:00:29.810 --> 01:00:32.720 align:middle line:84% in a slightly different way [INAUDIBLE].. 01:00:32.720 --> 01:00:36.310 align:middle line:84% This is the Syrian optic solution or the [INAUDIBLE],, 01:00:36.310 --> 01:00:40.450 align:middle line:84% which is actually bought as another company. 01:00:40.450 --> 01:00:43.930 align:middle line:84% [? Australia-- ?] forgetting the name. 01:00:43.930 --> 01:00:47.380 align:middle line:90% The solution is very similar. 01:00:47.380 --> 01:00:51.710 align:middle line:84% I'm sure they're fighting out in court right now. 01:00:51.710 --> 01:00:53.390 align:middle line:90% Same solution. 01:00:53.390 --> 01:00:56.060 align:middle line:84% Instead of putting this particular guy, 01:00:56.060 --> 01:01:03.510 align:middle line:84% that's just going to add some extra glass, but mostly 01:01:03.510 --> 01:01:04.200 align:middle line:90% in a minor form. 01:01:04.200 --> 01:01:07.170 align:middle line:90% 01:01:07.170 --> 01:01:10.470 align:middle line:84% It's just [INAUDIBLE] on that one. 01:01:10.470 --> 01:01:13.080 align:middle line:84% So basically the same solution but creating 01:01:13.080 --> 01:01:15.430 align:middle line:84% different focal length for different [? partners. ?] 01:01:15.430 --> 01:01:16.350 align:middle line:90% AUDIENCE: Yeah. 01:01:16.350 --> 01:01:18.930 align:middle line:84% Although you said, I mean, there's this portion there, 01:01:18.930 --> 01:01:22.585 align:middle line:84% where if you have another blur [INAUDIBLE],, right? 01:01:22.585 --> 01:01:23.460 align:middle line:90% RAMESH RASKAR: Right. 01:01:23.460 --> 01:01:26.710 align:middle line:84% AUDIENCE: But what is being blurred? 01:01:26.710 --> 01:01:28.833 align:middle line:90% At each piece, or where is? 01:01:28.833 --> 01:01:30.500 align:middle line:84% RAMESH RASKAR: Independent of the depth, 01:01:30.500 --> 01:01:32.310 align:middle line:90% you get the same blur. 01:01:32.310 --> 01:01:35.475 align:middle line:84% AUDIENCE: Yeah, but see, some guys are focusing, say-- 01:01:35.475 --> 01:01:36.600 align:middle line:90% RAMESH RASKAR: At an angle? 01:01:36.600 --> 01:01:38.138 align:middle line:90% It doesn't really matter. 01:01:38.138 --> 01:01:39.930 align:middle line:84% It doesn't really matter because, just like 01:01:39.930 --> 01:01:42.810 align:middle line:84% in a traditional camera, even if the point is not on axis 01:01:42.810 --> 01:01:46.190 align:middle line:84% but off axis, you still get the same-- you'll still get a disc, 01:01:46.190 --> 01:01:47.520 align:middle line:90% right, which we saw in the-- 01:01:47.520 --> 01:01:49.050 align:middle line:84% AUDIENCE: Yeah, you get the deuce, 01:01:49.050 --> 01:01:50.850 align:middle line:84% but I think that the given picture, 01:01:50.850 --> 01:01:53.070 align:middle line:84% as you just move it back and forth, 01:01:53.070 --> 01:01:55.150 align:middle line:84% you're going to get a different color, as you-- 01:01:55.150 --> 01:01:57.908 align:middle line:84% I mean, a different amount of the mixture of-- 01:01:57.908 --> 01:02:00.450 align:middle line:84% RAMESH RASKAR: Different shape, you mean, or different color? 01:02:00.450 --> 01:02:01.587 align:middle line:90% AUDIENCE: Different color. 01:02:01.587 --> 01:02:02.670 align:middle line:90% RAMESH RASKAR: Not really. 01:02:02.670 --> 01:02:03.510 align:middle line:90% AUDIENCE: Because look at that. 01:02:03.510 --> 01:02:05.280 align:middle line:84% If the guy was coming from the top, 01:02:05.280 --> 01:02:06.840 align:middle line:84% it's going to reach at some point. 01:02:06.840 --> 01:02:07.650 align:middle line:84% RAMESH RASKAR: But they all have the same-- 01:02:07.650 --> 01:02:09.840 align:middle line:84% I mean, you're saying, because of chromatic aberration? 01:02:09.840 --> 01:02:12.030 align:middle line:84% AUDIENCE: No, just because of the geometry, at least 01:02:12.030 --> 01:02:13.470 align:middle line:90% it seems to me. 01:02:13.470 --> 01:02:17.100 align:middle line:84% RAMESH RASKAR: Using color or shape? 01:02:17.100 --> 01:02:20.373 align:middle line:84% Because just to be clear that we are not adding any color here. 01:02:20.373 --> 01:02:21.540 align:middle line:90% We're just adding one glass. 01:02:21.540 --> 01:02:22.300 align:middle line:90% AUDIENCE: OK. 01:02:22.300 --> 01:02:23.050 align:middle line:90% OK, OK. 01:02:23.050 --> 01:02:24.970 align:middle line:84% RAMESH RASKAR: We're just adding one glass. 01:02:24.970 --> 01:02:27.382 align:middle line:84% So we're painting the rays, but the colors are, 01:02:27.382 --> 01:02:29.340 align:middle line:84% for all practical purposes, that'd be the same. 01:02:29.340 --> 01:02:32.610 align:middle line:84% AUDIENCE: Yeah, what I find is that maybe some points 01:02:32.610 --> 01:02:34.670 align:middle line:90% in the scene would be-- 01:02:34.670 --> 01:02:36.687 align:middle line:90% made sure even a piece of-- 01:02:36.687 --> 01:02:39.270 align:middle line:84% RAMESH RASKAR: Yeah, the effect is very low, though, remember. 01:02:39.270 --> 01:02:41.490 align:middle line:90% The effect is extremely low. 01:02:41.490 --> 01:02:43.960 align:middle line:84% So maybe you have a pixel and get blurred 01:02:43.960 --> 01:02:46.680 align:middle line:90% by 10 pixels or [INAUDIBLE]. 01:02:46.680 --> 01:02:48.760 align:middle line:90% It's not a global effect. 01:02:48.760 --> 01:02:51.090 align:middle line:84% So this picture, maybe-- this particular diagram 01:02:51.090 --> 01:02:53.430 align:middle line:84% is misleading because it seems like this point is 01:02:53.430 --> 01:02:54.750 align:middle line:90% going to go all the way. 01:02:54.750 --> 01:02:56.910 align:middle line:90% But this is very narrow. 01:02:56.910 --> 01:02:59.250 align:middle line:84% And the blur is only about 10 pixels, 01:02:59.250 --> 01:03:02.180 align:middle line:90% no matter where you [INAUDIBLE]. 01:03:02.180 --> 01:03:03.890 align:middle line:90% So maybe that was the matter. 01:03:03.890 --> 01:03:05.560 align:middle line:84% So if you have a point of access, 01:03:05.560 --> 01:03:09.760 align:middle line:84% it's still going to create an image that's blurred 10 pixels. 01:03:09.760 --> 01:03:12.730 align:middle line:84% So this is, again, very counterintuitive, 01:03:12.730 --> 01:03:17.110 align:middle line:84% where you go to make the image intentionally blurred. 01:03:17.110 --> 01:03:21.220 align:middle line:84% It's just that it's blurred everywhere. 01:03:21.220 --> 01:03:23.830 align:middle line:84% And then we also saw this one very early on, 01:03:23.830 --> 01:03:26.350 align:middle line:84% where the point spread function-- 01:03:26.350 --> 01:03:28.570 align:middle line:84% typically when something goes in and out of focus, 01:03:28.570 --> 01:03:29.885 align:middle line:90% it looks like a point. 01:03:29.885 --> 01:03:32.260 align:middle line:84% And then when it goes out of focus, it looks like a disc. 01:03:32.260 --> 01:03:33.718 align:middle line:84% If it goes out of focus other ways, 01:03:33.718 --> 01:03:35.080 align:middle line:90% it still looks like a disc. 01:03:35.080 --> 01:03:38.230 align:middle line:84% But this group at, again, at Colorado 01:03:38.230 --> 01:03:40.240 align:middle line:84% have-- when it's a sharp focus, you 01:03:40.240 --> 01:03:42.610 align:middle line:90% see two doors for similarity. 01:03:42.610 --> 01:03:44.920 align:middle line:84% And if you go in and out of focus, then 01:03:44.920 --> 01:03:48.870 align:middle line:90% the two dots [INAUDIBLE]. 01:03:48.870 --> 01:03:51.058 align:middle line:84% So they call it rotating point spread function. 01:03:51.058 --> 01:03:53.100 align:middle line:84% AUDIENCE: Is it the same group that developed the 01:03:53.100 --> 01:03:55.080 align:middle line:90% [? framework? ?] 01:03:55.080 --> 01:03:57.840 align:middle line:84% RAMESH RASKAR: It's not the same group, but same university 01:03:57.840 --> 01:04:00.450 align:middle line:90% and the same neighborhood. 01:04:00.450 --> 01:04:03.210 align:middle line:84% AUDIENCE: What was the reasoning for developing the rotating 01:04:03.210 --> 01:04:05.520 align:middle line:90% point spread function? 01:04:05.520 --> 01:04:08.640 align:middle line:84% RAMESH RASKAR: Doug's question is, what's the benefit of this? 01:04:08.640 --> 01:04:09.952 align:middle line:90% AUDIENCE: Does [INAUDIBLE]? 01:04:09.952 --> 01:04:11.865 align:middle line:90% [LAUGHTER] 01:04:11.865 --> 01:04:13.740 align:middle line:84% RAMESH RASKAR: She would have used it by now. 01:04:13.740 --> 01:04:14.365 align:middle line:90% AUDIENCE: Yeah. 01:04:14.365 --> 01:04:16.615 align:middle line:84% RAMESH RASKAR: What's the benefit of the strange point 01:04:16.615 --> 01:04:17.430 align:middle line:90% spread function? 01:04:17.430 --> 01:04:19.470 align:middle line:84% AUDIENCE: You know if you're out of focus in which direction. 01:04:19.470 --> 01:04:19.920 align:middle line:90% AUDIENCE: Yeah. 01:04:19.920 --> 01:04:20.220 align:middle line:90% RAMESH RASKAR: Yeah? 01:04:20.220 --> 01:04:20.910 align:middle line:90% AUDIENCE: From the focal point. 01:04:20.910 --> 01:04:21.840 align:middle line:90% RAMESH RASKAR: That's one. 01:04:21.840 --> 01:04:22.740 align:middle line:90% AUDIENCE: And you know your-- 01:04:22.740 --> 01:04:23.880 align:middle line:84% RAMESH RASKAR: But do you know by how much? 01:04:23.880 --> 01:04:25.230 align:middle line:84% AUDIENCE: Yeah, you know by how much because-- 01:04:25.230 --> 01:04:26.250 align:middle line:84% RAMESH RASKAR: Because it's an angle. 01:04:26.250 --> 01:04:27.125 align:middle line:90% AUDIENCE: --rotation. 01:04:27.125 --> 01:04:31.650 align:middle line:84% RAMESH RASKAR: So the goal here was no matter where you are, 01:04:31.650 --> 01:04:34.320 align:middle line:84% your point spread function is the same. 01:04:34.320 --> 01:04:35.880 align:middle line:84% The goal here is exactly opposite. 01:04:35.880 --> 01:04:38.910 align:middle line:84% If you go slightly out of focus, you get a very different point 01:04:38.910 --> 01:04:40.360 align:middle line:90% spread function. 01:04:40.360 --> 01:04:42.930 align:middle line:84% So this one they use in microscopy 01:04:42.930 --> 01:04:44.130 align:middle line:90% with fluorescent dye. 01:04:44.130 --> 01:04:45.900 align:middle line:84% So when you're looking with a microscope, 01:04:45.900 --> 01:04:51.360 align:middle line:84% depending on what the depth of your tagged particle is, 01:04:51.360 --> 01:04:53.950 align:middle line:84% the point spread function will look very different. 01:04:53.950 --> 01:04:56.640 align:middle line:84% So you can estimate the depth by looking at the orientation 01:04:56.640 --> 01:04:59.090 align:middle line:90% of those two dots. 01:04:59.090 --> 01:05:03.180 align:middle line:90% So that's very interesting. 01:05:03.180 --> 01:05:05.972 align:middle line:84% AUDIENCE: But can't that guy keep going all the way in? 01:05:05.972 --> 01:05:07.055 align:middle line:90% At some point, you can't-- 01:05:07.055 --> 01:05:08.180 align:middle line:84% RAMESH RASKAR: No, it doesn't work. 01:05:08.180 --> 01:05:10.140 align:middle line:84% After some point, they'll stay the same. 01:05:10.140 --> 01:05:11.300 align:middle line:90% AUDIENCE: OK. 01:05:11.300 --> 01:05:13.550 align:middle line:84% RAMESH RASKAR: This is only in the [? sweet ?] region. 01:05:13.550 --> 01:05:16.390 align:middle line:84% AUDIENCE: So have they been able to reconstruct 01:05:16.390 --> 01:05:18.440 align:middle line:84% three-dimensional neuronal structures, or-- 01:05:18.440 --> 01:05:21.065 align:middle line:84% RAMESH RASKAR: Yeah, that's why they're getting a lot of press. 01:05:21.065 --> 01:05:24.920 align:middle line:84% And they're doing some amazing work, [INAUDIBLE].. 01:05:24.920 --> 01:05:27.020 align:middle line:84% So they have a lot of collaborations, 01:05:27.020 --> 01:05:32.270 align:middle line:84% and now they're able to measure the z-dimension down 01:05:32.270 --> 01:05:33.500 align:middle line:90% to about 10 nanometers. 01:05:33.500 --> 01:05:34.760 align:middle line:90% AUDIENCE: Wow. 01:05:34.760 --> 01:05:37.940 align:middle line:84% RAMESH RASKAR: The xy still remains traditional microscope 01:05:37.940 --> 01:05:39.710 align:middle line:90% 1 micron, 1/2 micron. 01:05:39.710 --> 01:05:42.320 align:middle line:84% But the z-dimension is 10 nanometers. 01:05:42.320 --> 01:05:45.117 align:middle line:90% 01:05:45.117 --> 01:05:45.700 align:middle line:90% It's very new. 01:05:45.700 --> 01:05:49.040 align:middle line:84% They are still working on a lot of these concepts. 01:05:49.040 --> 01:05:49.550 align:middle line:90% OK? 01:05:49.550 --> 01:05:54.200 align:middle line:84% So let's very briefly look at compressed sensing 01:05:54.200 --> 01:05:57.380 align:middle line:84% because it's something you should be familiar with. 01:05:57.380 --> 01:06:05.930 align:middle line:90% 01:06:05.930 --> 01:06:10.640 align:middle line:84% OK, so here's an idea that received a lot of publicity. 01:06:10.640 --> 01:06:14.300 align:middle line:84% It was even "The 10 Emerging Technologies" 01:06:14.300 --> 01:06:17.180 align:middle line:90% by a very reputable magazine. 01:06:17.180 --> 01:06:20.173 align:middle line:84% I hope you don't believe any of those things. 01:06:20.173 --> 01:06:21.590 align:middle line:84% It's a very cool idea, by the way. 01:06:21.590 --> 01:06:23.510 align:middle line:84% And as a scientist, I really like it. 01:06:23.510 --> 01:06:27.770 align:middle line:84% But when somebody like Technology Review or Wired 01:06:27.770 --> 01:06:32.850 align:middle line:84% Magazine says, Top 50, Top 10, of course, 01:06:32.850 --> 01:06:34.130 align:middle line:90% I wish I'm listed among them. 01:06:34.130 --> 01:06:36.560 align:middle line:84% But at the same time-- because, you know, 01:06:36.560 --> 01:06:40.200 align:middle line:90% it has good side effects. 01:06:40.200 --> 01:06:41.870 align:middle line:84% Well, anyway, this single-pixel camera 01:06:41.870 --> 01:06:45.890 align:middle line:84% was listed as one of the big things in 2005 01:06:45.890 --> 01:06:48.170 align:middle line:84% by Technology Review, which a magazine I really like, 01:06:48.170 --> 01:06:49.610 align:middle line:90% by the way. 01:06:49.610 --> 01:06:52.100 align:middle line:84% And the idea is, instead of taking one 01:06:52.100 --> 01:06:55.520 align:middle line:84% single photo, what you're going to do is-- 01:06:55.520 --> 01:06:56.780 align:middle line:90% let's say that's your scene. 01:06:56.780 --> 01:06:58.340 align:middle line:90% You're go to turn on-- 01:06:58.340 --> 01:07:01.010 align:middle line:84% you go to take a single photodetector 01:07:01.010 --> 01:07:05.570 align:middle line:84% and aim it at a set of micrometers. 01:07:05.570 --> 01:07:07.610 align:middle line:84% And in the simplest case, what you 01:07:07.610 --> 01:07:10.390 align:middle line:84% will do is you turn off all the micrometers, 01:07:10.390 --> 01:07:12.110 align:middle line:90% that light goes this way. 01:07:12.110 --> 01:07:15.440 align:middle line:84% And there's only one micrometer, like this one. 01:07:15.440 --> 01:07:18.260 align:middle line:84% So a single photodetector-- this is like the dual photography 01:07:18.260 --> 01:07:22.370 align:middle line:84% we saw right at the beginning, where you can see that card. 01:07:22.370 --> 01:07:25.340 align:middle line:84% If I just turn on this one micrometer-- by the way, 01:07:25.340 --> 01:07:28.010 align:middle line:84% this is what's in your DLP projectors, 01:07:28.010 --> 01:07:31.670 align:middle line:84% the Texas Instruments Digital Light Processing Micrometer 01:07:31.670 --> 01:07:32.600 align:middle line:90% Displays. 01:07:32.600 --> 01:07:35.180 align:middle line:90% So it's very easily available. 01:07:35.180 --> 01:07:40.110 align:middle line:84% | just receive light from the scene for that one pixel. 01:07:40.110 --> 01:07:43.430 align:middle line:84% So this scene is being imaged on this little array. 01:07:43.430 --> 01:07:45.500 align:middle line:84% And you just want to turn on this one pixel. 01:07:45.500 --> 01:07:47.030 align:middle line:84% And then the next picture, you're 01:07:47.030 --> 01:07:49.200 align:middle line:84% going to turn on the next pixel, and so on. 01:07:49.200 --> 01:07:52.400 align:middle line:84% And one at a time, if you go through this million pixels, 01:07:52.400 --> 01:07:57.980 align:middle line:84% you will get a million megapixel image, right? 01:07:57.980 --> 01:08:00.290 align:middle line:84% But of course, the light will be very little 01:08:00.290 --> 01:08:01.770 align:middle line:90% if you just turn on one pixel. 01:08:01.770 --> 01:08:04.580 align:middle line:84% So now, we'll do some [INAUDIBLE] multiplexing, 01:08:04.580 --> 01:08:06.920 align:middle line:84% which we saw a few classes ago, where 01:08:06.920 --> 01:08:11.320 align:middle line:84% you go to turn on over half of them, take one reading, 01:08:11.320 --> 01:08:13.910 align:middle line:84% turn some other random combination of half of them, 01:08:13.910 --> 01:08:15.600 align:middle line:90% and take a picture, and so on. 01:08:15.600 --> 01:08:17.930 align:middle line:84% And, again, after-- now about half of them 01:08:17.930 --> 01:08:19.500 align:middle line:84% are contributing to the photodiodes. 01:08:19.500 --> 01:08:21.852 align:middle line:84% So the photodiode is very well exposed, 01:08:21.852 --> 01:08:23.810 align:middle line:84% and you can take a very short exposure reading. 01:08:23.810 --> 01:08:25.910 align:middle line:84% And, again, if you take million such readings, 01:08:25.910 --> 01:08:28.880 align:middle line:90% you can recover this picture. 01:08:28.880 --> 01:08:30.320 align:middle line:90% That's the concept. 01:08:30.320 --> 01:08:33.913 align:middle line:90% 01:08:33.913 --> 01:08:36.080 align:middle line:84% AUDIENCE: Does it exponentially increase, the number 01:08:36.080 --> 01:08:37.010 align:middle line:90% of readings you have to take? 01:08:37.010 --> 01:08:37.880 align:middle line:84% RAMESH RASKAR: No, just linearly. 01:08:37.880 --> 01:08:40.297 align:middle line:84% If you're on 2 megapixels, then you need to take 2 million 01:08:40.297 --> 01:08:42.224 align:middle line:90% [? pics. ?] All right? 01:08:42.224 --> 01:08:45.939 align:middle line:90% 01:08:45.939 --> 01:08:49.779 align:middle line:84% So the claim this group made at Rice University 01:08:49.779 --> 01:08:53.290 align:middle line:84% was that if I wanted a million-pixel image, 01:08:53.290 --> 01:08:56.020 align:middle line:84% I don't have to really take a million readings. 01:08:56.020 --> 01:09:01.270 align:middle line:84% I can do much fewer than a million readings. 01:09:01.270 --> 01:09:07.330 align:middle line:84% And the claim is that imagine if you had this photo as a JPEG. 01:09:07.330 --> 01:09:10.390 align:middle line:84% In a composite, it might take up only about tens 01:09:10.390 --> 01:09:11.870 align:middle line:90% of thousands of bytes. 01:09:11.870 --> 01:09:14.029 align:middle line:84% So let's say it takes up 10,000 bytes. 01:09:14.029 --> 01:09:18.609 align:middle line:84% So if I can represent the image with 10,000 bytes, 01:09:18.609 --> 01:09:20.890 align:middle line:84% and I'm going to take a photo and compress it down 01:09:20.890 --> 01:09:23.740 align:middle line:84% to 10,000 bytes, I can't just directly measure 01:09:23.740 --> 01:09:29.859 align:middle line:84% only 10,000 values in the scene so that I save on everything, 01:09:29.859 --> 01:09:30.370 align:middle line:90% OK? 01:09:30.370 --> 01:09:32.140 align:middle line:84% So I can take this picture effectively 01:09:32.140 --> 01:09:37.700 align:middle line:84% with just 10,000 pixels but recreate a million-pixel image. 01:09:37.700 --> 01:09:40.660 align:middle line:84% And that's where the concept of compressive sensing 01:09:40.660 --> 01:09:43.450 align:middle line:90% or compressed imaging comes up. 01:09:43.450 --> 01:09:47.350 align:middle line:84% You want to take something that is much higher resolution 01:09:47.350 --> 01:09:50.140 align:middle line:84% but recover it in a compressed way, where 01:09:50.140 --> 01:09:53.260 align:middle line:84% it's taking the picture with a hardware 01:09:53.260 --> 01:09:54.880 align:middle line:90% and compressing the software. 01:09:54.880 --> 01:09:59.080 align:middle line:84% You're going to compress it while sensing. 01:09:59.080 --> 01:10:03.530 align:middle line:84% So how does it look mathematically? 01:10:03.530 --> 01:10:06.046 align:middle line:90% So let's see. 01:10:06.046 --> 01:10:07.870 align:middle line:84% Let's see if there's an easy way to explain 01:10:07.870 --> 01:10:09.760 align:middle line:90% this in a shorter time. 01:10:09.760 --> 01:10:12.490 align:middle line:90% 01:10:12.490 --> 01:10:14.423 align:middle line:84% So that's the trick we're going to do. 01:10:14.423 --> 01:10:16.090 align:middle line:84% We're going to take about half the pixel 01:10:16.090 --> 01:10:18.980 align:middle line:84% and measure the intensity and so on. 01:10:18.980 --> 01:10:20.510 align:middle line:90% So those are our measurements. 01:10:20.510 --> 01:10:23.595 align:middle line:90% So our unknown image is x. 01:10:23.595 --> 01:10:25.720 align:middle line:84% And we're going to take a lot of these projections. 01:10:25.720 --> 01:10:28.548 align:middle line:84% This is the [INAUDIBLE] matrix, for those of you familiar. 01:10:28.548 --> 01:10:29.840 align:middle line:90% And these are our measurements. 01:10:29.840 --> 01:10:32.110 align:middle line:84% So we're going to say, given these measurements, 01:10:32.110 --> 01:10:37.400 align:middle line:84% I'm going to recover my original image. 01:10:37.400 --> 01:10:42.750 align:middle line:84% Now, when you think about a natural image, 01:10:42.750 --> 01:10:46.160 align:middle line:84% the claim is that if you just use DCT, 01:10:46.160 --> 01:10:49.670 align:middle line:84% some photo coefficients, then you can compress the image 01:10:49.670 --> 01:10:53.120 align:middle line:84% and represent them with very few bytes, only 01:10:53.120 --> 01:10:55.460 align:middle line:90% 10,000 bytes for a megapixel. 01:10:55.460 --> 01:11:00.720 align:middle line:84% So let's say your Fourier coefficients are here. 01:11:00.720 --> 01:11:02.970 align:middle line:90% And this is your image. 01:11:02.970 --> 01:11:06.260 align:middle line:84% That means that if I just put a Fourier transform here, 01:11:06.260 --> 01:11:09.920 align:middle line:84% then I can convert the coefficients into the image. 01:11:09.920 --> 01:11:15.980 align:middle line:84% And the number of values required to represent an image 01:11:15.980 --> 01:11:19.650 align:middle line:84% are much fewer than the million values required here. 01:11:19.650 --> 01:11:22.700 align:middle line:84% So we have a million values here, but only about 10,000 01:11:22.700 --> 01:11:24.350 align:middle line:90% values here. 01:11:24.350 --> 01:11:28.340 align:middle line:84% And the claim is that by using this understanding that 01:11:28.340 --> 01:11:32.030 align:middle line:84% my image can be represented in some transform basis-- 01:11:32.030 --> 01:11:33.500 align:middle line:90% in this case, Fourier basis-- 01:11:33.500 --> 01:11:38.420 align:middle line:84% using very few coefficients, can be exploited while I'm sensing. 01:11:38.420 --> 01:11:40.520 align:middle line:90% OK, this is your optics. 01:11:40.520 --> 01:11:41.240 align:middle line:90% This is your map. 01:11:41.240 --> 01:11:44.150 align:middle line:90% 01:11:44.150 --> 01:11:47.570 align:middle line:90% See if I have it on slide 5. 01:11:47.570 --> 01:11:52.250 align:middle line:84% So that's the theory of compressive sensing, 01:11:52.250 --> 01:11:56.880 align:middle line:84% that, using some basis, I can transform the image and measure 01:11:56.880 --> 01:11:59.390 align:middle line:90% in [? your ?] measurements. 01:11:59.390 --> 01:12:02.000 align:middle line:84% And there are certain cases where it is really true. 01:12:02.000 --> 01:12:06.230 align:middle line:84% You have signals that can be compressed very easily. 01:12:06.230 --> 01:12:09.680 align:middle line:84% A very classic example is in communication, 01:12:09.680 --> 01:12:13.250 align:middle line:84% where, if you are doing software radio, where you have 01:12:13.250 --> 01:12:19.130 align:middle line:84% a huge band of frequencies, and software radio-- instead 01:12:19.130 --> 01:12:22.790 align:middle line:84% of tuning it with electromagnetics, 01:12:22.790 --> 01:12:24.380 align:middle line:84% you just capture the whole signal. 01:12:24.380 --> 01:12:27.080 align:middle line:84% And then software, you can listen to any station. 01:12:27.080 --> 01:12:29.690 align:middle line:90% 01:12:29.690 --> 01:12:34.790 align:middle line:84% And the necklace theory says, if your band is, I don't know, 01:12:34.790 --> 01:12:37.430 align:middle line:84% 100 megahertz, then you must capture it 01:12:37.430 --> 01:12:41.780 align:middle line:84% with a signal that has a bandwidth of 100 milliamps. 01:12:41.780 --> 01:12:45.363 align:middle line:84% But we know that in communication, not all bands 01:12:45.363 --> 01:12:46.280 align:middle line:90% are actually occupied. 01:12:46.280 --> 01:12:48.530 align:middle line:90% Many of the bands are empty. 01:12:48.530 --> 01:12:51.020 align:middle line:84% Only certain frequencies have a signal. 01:12:51.020 --> 01:12:53.840 align:middle line:84% So people have come up with very clever mechanisms, where 01:12:53.840 --> 01:12:56.220 align:middle line:84% they realize that you don't have to capture 01:12:56.220 --> 01:12:58.700 align:middle line:90% a 100-megahertz signal. 01:12:58.700 --> 01:13:00.950 align:middle line:84% Only some of them are actually on. 01:13:00.950 --> 01:13:02.720 align:middle line:84% I'm getting through the Fourier transform 01:13:02.720 --> 01:13:06.050 align:middle line:84% because in communication, that's natural. 01:13:06.050 --> 01:13:08.060 align:middle line:84% And by doing that, they're able to sample 01:13:08.060 --> 01:13:13.910 align:middle line:84% this effect of a software radio with a detector that 01:13:13.910 --> 01:13:17.900 align:middle line:84% doesn't have to measure 100-megahertz-wide signal. 01:13:17.900 --> 01:13:22.370 align:middle line:84% It turns out for images, this doesn't work. 01:13:22.370 --> 01:13:27.500 align:middle line:84% And that's because there is no transform that 01:13:27.500 --> 01:13:29.900 align:middle line:84% allows you-- no linear transform that 01:13:29.900 --> 01:13:34.190 align:middle line:84% allows you to represent an image with very few coefficients. 01:13:34.190 --> 01:13:37.880 align:middle line:84% When you do JPEG, it does frequency transform. 01:13:37.880 --> 01:13:39.860 align:middle line:84% But after that, it does a lot of other things. 01:13:39.860 --> 01:13:42.770 align:middle line:84% It says, perceptually, the higher frequencies 01:13:42.770 --> 01:13:45.110 align:middle line:84% are not as important, so I'm going to represent them 01:13:45.110 --> 01:13:46.850 align:middle line:90% with fewer quantization grids. 01:13:46.850 --> 01:13:49.100 align:middle line:90% Or certain values are too small. 01:13:49.100 --> 01:13:50.450 align:middle line:90% I'm just going to truncate them. 01:13:50.450 --> 01:13:53.810 align:middle line:84% So all this operation-- changing quantization bands, 01:13:53.810 --> 01:13:57.530 align:middle line:84% truncating, or thresholding, are all nonlinear operations. 01:13:57.530 --> 01:13:59.000 align:middle line:90% They are not linear operations. 01:13:59.000 --> 01:14:01.070 align:middle line:84% So it turns out there is no transform that 01:14:01.070 --> 01:14:04.950 align:middle line:84% allows you to represent an image with fewer coefficients. 01:14:04.950 --> 01:14:08.450 align:middle line:84% So in general, this scheme doesn't work. 01:14:08.450 --> 01:14:11.220 align:middle line:84% But you will continue to see people who come to you and say, 01:14:11.220 --> 01:14:13.830 align:middle line:84% you know, I have this magical thing 01:14:13.830 --> 01:14:17.540 align:middle line:84% I just heard or compressive image something, 01:14:17.540 --> 01:14:20.120 align:middle line:84% and that will just solve a problem. 01:14:20.120 --> 01:14:22.610 align:middle line:84% There are certain images, like cartoons, 01:14:22.610 --> 01:14:24.770 align:middle line:84% that can be represented with very few samples 01:14:24.770 --> 01:14:27.050 align:middle line:84% because they have flat regions, sharp boundaries, 01:14:27.050 --> 01:14:28.160 align:middle line:90% and fluctuations. 01:14:28.160 --> 01:14:29.900 align:middle line:84% But a natural image, unfortunately, 01:14:29.900 --> 01:14:32.000 align:middle line:84% cannot be transformed that easily. 01:14:32.000 --> 01:14:33.740 align:middle line:84% And you can talk to Rohit, and he'll 01:14:33.740 --> 01:14:38.900 align:middle line:84% tell you all the details of the dangers of [? compositions. ?] 01:14:38.900 --> 01:14:41.870 align:middle line:84% AUDIENCE: So the single-pixel camera is just a hypothesis 01:14:41.870 --> 01:14:42.800 align:middle line:90% but not-- 01:14:42.800 --> 01:14:44.550 align:middle line:84% RAMESH RASKAR: Yeah, but at the same time, 01:14:44.550 --> 01:14:49.610 align:middle line:84% it was the first one that kind of allowed people to visualize 01:14:49.610 --> 01:14:52.460 align:middle line:84% or kind of conceptualize in their mind 01:14:52.460 --> 01:14:54.200 align:middle line:84% what compressive sensing might do. 01:14:54.200 --> 01:14:57.340 align:middle line:90% 01:14:57.340 --> 01:15:00.370 align:middle line:84% AUDIENCE: This idea is cool, but how feasible 01:15:00.370 --> 01:15:03.370 align:middle line:84% or how important it is to have a single sensor rather 01:15:03.370 --> 01:15:05.380 align:middle line:90% than having wide arrows sensing? 01:15:05.380 --> 01:15:07.540 align:middle line:84% So what this is achieving is basically 01:15:07.540 --> 01:15:10.600 align:middle line:84% allowing you to build a camera with a single sensor. 01:15:10.600 --> 01:15:15.717 align:middle line:84% But do we really want it just to do compressed sensing? 01:15:15.717 --> 01:15:17.300 align:middle line:84% RAMESH RASKAR: From a scientific point 01:15:17.300 --> 01:15:20.000 align:middle line:84% of view, if somebody can build this and show that you can 01:15:20.000 --> 01:15:22.850 align:middle line:84% take fewer measurements and recover the image, 01:15:22.850 --> 01:15:25.250 align:middle line:90% that's a breakthrough. 01:15:25.250 --> 01:15:26.120 align:middle line:90% How do you use it? 01:15:26.120 --> 01:15:29.250 align:middle line:84% I agree with you that, in terms of practical implementation, 01:15:29.250 --> 01:15:32.910 align:middle line:84% maybe this is the best application, maybe it's not, 01:15:32.910 --> 01:15:33.930 align:middle line:90% and so on. 01:15:33.930 --> 01:15:36.980 align:middle line:84% But that's kind of a business reason. 01:15:36.980 --> 01:15:42.440 align:middle line:84% AUDIENCE: Will this be faster than an array of sensors? 01:15:42.440 --> 01:15:46.410 align:middle line:84% RAMESH RASKAR: Again, in terms of in practice, both of you 01:15:46.410 --> 01:15:46.910 align:middle line:90% are right. 01:15:46.910 --> 01:15:48.077 align:middle line:90% There are very few benefits. 01:15:48.077 --> 01:15:51.630 align:middle line:90% 01:15:51.630 --> 01:15:54.530 align:middle line:84% But if you just do compressive sensing, 01:15:54.530 --> 01:15:58.860 align:middle line:84% you realize it's a very, very active field. 01:15:58.860 --> 01:16:03.410 align:middle line:84% AUDIENCE: So again, maybe a different type of sensing, 01:16:03.410 --> 01:16:08.090 align:middle line:90% but what are the features-- 01:16:08.090 --> 01:16:10.590 align:middle line:84% like, what are the people doing in computational photography 01:16:10.590 --> 01:16:12.330 align:middle line:84% for feature extraction in the same way 01:16:12.330 --> 01:16:15.435 align:middle line:84% that the brain processes certain features of linear 01:16:15.435 --> 01:16:21.750 align:middle line:84% [INAUDIBLE] to do a better compress sensing 01:16:21.750 --> 01:16:24.427 align:middle line:90% of context and imaging? 01:16:24.427 --> 01:16:26.260 align:middle line:84% RAMESH RASKAR: You mean compressing an image 01:16:26.260 --> 01:16:27.568 align:middle line:90% or sensing with fewer samples? 01:16:27.568 --> 01:16:29.110 align:middle line:84% AUDIENCE: Sensing with fewer samples. 01:16:29.110 --> 01:16:31.390 align:middle line:84% RAMESH RASKAR: Yeah, so that all kind of gets clubbed 01:16:31.390 --> 01:16:33.098 align:middle line:84% into this concept of compressive sensing. 01:16:33.098 --> 01:16:36.400 align:middle line:90% 01:16:36.400 --> 01:16:39.665 align:middle line:84% If you think about a B1 and B2 and visual processing, 01:16:39.665 --> 01:16:42.040 align:middle line:84% there's a lot of work that has been done over the last 30 01:16:42.040 --> 01:16:42.540 align:middle line:90% years. 01:16:42.540 --> 01:16:45.580 align:middle line:84% There's good work at CSAIL as well. 01:16:45.580 --> 01:16:47.463 align:middle line:90% But that's purely software. 01:16:47.463 --> 01:16:48.130 align:middle line:90% AUDIENCE: Right. 01:16:48.130 --> 01:16:49.780 align:middle line:84% RAMESH RASKAR: And maybe you're asking, 01:16:49.780 --> 01:16:54.670 align:middle line:84% can we use sensing mechanisms that are similar to our brain 01:16:54.670 --> 01:16:55.745 align:middle line:90% so that we don't-- 01:16:55.745 --> 01:16:57.370 align:middle line:84% AUDIENCE: You don't to do any software. 01:16:57.370 --> 01:16:58.328 align:middle line:90% RAMESH RASKAR: Exactly. 01:16:58.328 --> 01:17:02.500 align:middle line:84% The secret of success for film, of film photography, 01:17:02.500 --> 01:17:05.830 align:middle line:84% is that if somebody had given you 01:17:05.830 --> 01:17:08.800 align:middle line:84% this problem before the invention of film, 01:17:08.800 --> 01:17:10.420 align:middle line:90% that there is a scene-- 01:17:10.420 --> 01:17:15.070 align:middle line:84% and I want to give you a sensation of the same scene-- 01:17:15.070 --> 01:17:18.650 align:middle line:90% time shifted or space shifted. 01:17:18.650 --> 01:17:20.650 align:middle line:84% There's so many ways you can solve that problem. 01:17:20.650 --> 01:17:25.540 align:middle line:84% You can start with a reproduction of a photo, 01:17:25.540 --> 01:17:27.220 align:middle line:90% or you can tap into retina. 01:17:27.220 --> 01:17:28.660 align:middle line:90% You can tap into V1, V2. 01:17:28.660 --> 01:17:32.110 align:middle line:84% You can interface that at any point in the pipeline 01:17:32.110 --> 01:17:35.080 align:middle line:90% for human vision. 01:17:35.080 --> 01:17:38.800 align:middle line:84% But the simplest solution is to just create 01:17:38.800 --> 01:17:44.590 align:middle line:84% that photo on a passive surface and let the brain do 01:17:44.590 --> 01:17:47.360 align:middle line:90% that processing all over again. 01:17:47.360 --> 01:17:49.150 align:middle line:84% And so it's like a simple impedance match. 01:17:49.150 --> 01:17:51.040 align:middle line:84% If I can see the scene and understand it, 01:17:51.040 --> 01:17:54.818 align:middle line:84% I can just present that as is and let it go through. 01:17:54.818 --> 01:17:56.860 align:middle line:84% And this is how we have been treating photography 01:17:56.860 --> 01:17:57.580 align:middle line:90% all this time. 01:17:57.580 --> 01:18:01.030 align:middle line:84% It's a record of visual experience, which 01:18:01.030 --> 01:18:04.052 align:middle line:84% is great for humans, but it's not so great for computers 01:18:04.052 --> 01:18:06.010 align:middle line:84% because computers don't understand any of that. 01:18:06.010 --> 01:18:08.590 align:middle line:84% And what you're saying is, what computers care about 01:18:08.590 --> 01:18:10.953 align:middle line:84% are all these high-level features. 01:18:10.953 --> 01:18:13.120 align:middle line:84% And that's why we're going back to the drawing board 01:18:13.120 --> 01:18:15.070 align:middle line:84% and saying, let's build cameras that are not 01:18:15.070 --> 01:18:18.280 align:middle line:84% mimicking human eye but actually extracting more information, 01:18:18.280 --> 01:18:21.010 align:middle line:84% like [? apertures ?] that we remove the flash camera, 01:18:21.010 --> 01:18:24.280 align:middle line:84% or additional information with light-field cameras 01:18:24.280 --> 01:18:26.268 align:middle line:84% or multi-spectral cameras and so on. 01:18:26.268 --> 01:18:29.690 align:middle line:90% 01:18:29.690 --> 01:18:31.822 align:middle line:84% AUDIENCE: So what kind of-- so Brett was asking, 01:18:31.822 --> 01:18:33.530 align:middle line:84% why would you want to do [? precisely? ?] 01:18:33.530 --> 01:18:36.320 align:middle line:84% When do you have to reduce the number of measurements 01:18:36.320 --> 01:18:37.620 align:middle line:90% [INAUDIBLE]? 01:18:37.620 --> 01:18:41.630 align:middle line:84% And I think one of the problem [INAUDIBLE].. 01:18:41.630 --> 01:18:42.178 align:middle line:90% I don't know. 01:18:42.178 --> 01:18:43.970 align:middle line:84% The debate about whether it's really better 01:18:43.970 --> 01:18:46.100 align:middle line:90% or not is photography? 01:18:46.100 --> 01:18:46.700 align:middle line:90% [INAUDIBLE] 01:18:46.700 --> 01:18:47.480 align:middle line:90% RAMESH RASKAR: Tomography, yeah. 01:18:47.480 --> 01:18:49.760 align:middle line:84% AUDIENCE: Yeah, when you have to recover the sky, 01:18:49.760 --> 01:18:52.290 align:middle line:84% you want to take as few measurements as possible. 01:18:52.290 --> 01:18:54.605 align:middle line:84% So if you can reduce that [INAUDIBLE],, 01:18:54.605 --> 01:18:56.480 align:middle line:84% That's one of the [? implications ?] of that. 01:18:56.480 --> 01:18:56.780 align:middle line:90% RAMESH RASKAR: Right. 01:18:56.780 --> 01:18:58.280 align:middle line:84% AUDIENCE: This one piece of camera-- 01:18:58.280 --> 01:18:59.940 align:middle line:90% really, it does that. 01:18:59.940 --> 01:19:02.630 align:middle line:84% RAMESH RASKAR: But the benefit of tomography, 01:19:02.630 --> 01:19:04.670 align:middle line:84% which we studied in the last couple of lectures, 01:19:04.670 --> 01:19:07.520 align:middle line:84% is it's a very high-dimensional signal. 01:19:07.520 --> 01:19:09.530 align:middle line:84% And so usually, in a high-dimensional signal, 01:19:09.530 --> 01:19:10.540 align:middle line:90% there's lower sparsity. 01:19:10.540 --> 01:19:11.930 align:middle line:90% There are only a few places. 01:19:11.930 --> 01:19:15.073 align:middle line:84% If you think about taking a CAT scan of your body, 01:19:15.073 --> 01:19:17.240 align:middle line:84% there are only like four or five types of materials. 01:19:17.240 --> 01:19:18.110 align:middle line:90% There is muscle. 01:19:18.110 --> 01:19:20.880 align:middle line:90% There is blood. 01:19:20.880 --> 01:19:21.380 align:middle line:90% Whatever. 01:19:21.380 --> 01:19:23.390 align:middle line:84% There are only five or six things. 01:19:23.390 --> 01:19:24.650 align:middle line:90% It's like a cartoon. 01:19:24.650 --> 01:19:25.400 align:middle line:90% AUDIENCE: Exactly. 01:19:25.400 --> 01:19:27.775 align:middle line:84% I find this is interesting because the test-- 01:19:27.775 --> 01:19:29.150 align:middle line:84% RAMESH RASKAR: It's a 3D cartoon. 01:19:29.150 --> 01:19:31.692 align:middle line:84% AUDIENCE: But if you look at it, it looks just like a cartoon 01:19:31.692 --> 01:19:34.072 align:middle line:84% does-- some whites clothes, some black clothes, some-- 01:19:34.072 --> 01:19:35.030 align:middle line:90% RAMESH RASKAR: Exactly. 01:19:35.030 --> 01:19:37.405 align:middle line:84% And that's why compressive sensing works very well there. 01:19:37.405 --> 01:19:39.820 align:middle line:90% 01:19:39.820 --> 01:19:42.070 align:middle line:84% AUDIENCE: So is compressive something used in anything 01:19:42.070 --> 01:19:44.170 align:middle line:90% commercially currently? 01:19:44.170 --> 01:19:46.400 align:middle line:84% RAMESH RASKAR: A lot of people are getting grants. 01:19:46.400 --> 01:19:46.942 align:middle line:90% AUDIENCE: Oh. 01:19:46.942 --> 01:19:48.220 align:middle line:90% [LAUGHTER] 01:19:48.220 --> 01:19:50.303 align:middle line:84% RAMESH RASKAR: Is that a commercial-enough reason? 01:19:50.303 --> 01:19:50.920 align:middle line:90% AUDIENCE: No. 01:19:50.920 --> 01:19:53.200 align:middle line:90% But also-- 01:19:53.200 --> 01:19:55.330 align:middle line:84% RAMESH RASKAR: If you put those two words, 01:19:55.330 --> 01:19:57.010 align:middle line:90% your chances improve by 50%. 01:19:57.010 --> 01:19:59.560 align:middle line:84% AUDIENCE: I was thinking of this interesting album that, 01:19:59.560 --> 01:20:01.685 align:middle line:84% I guess, extends to all you guys are talking about. 01:20:01.685 --> 01:20:04.750 align:middle line:84% So compressive sensing allows you to take less measurements. 01:20:04.750 --> 01:20:07.540 align:middle line:84% But the problem is you need to actually have more information 01:20:07.540 --> 01:20:10.210 align:middle line:84% about the scene before you take the measurement, which 01:20:10.210 --> 01:20:11.375 align:middle line:90% is another measurement. 01:20:11.375 --> 01:20:13.000 align:middle line:84% RAMESH RASKAR: So actually, to clarify, 01:20:13.000 --> 01:20:15.730 align:middle line:84% the measurements are done in a non-adaptive manner. 01:20:15.730 --> 01:20:18.220 align:middle line:84% So you don't have to know anything about the scene 01:20:18.220 --> 01:20:19.390 align:middle line:90% to do this measurements. 01:20:19.390 --> 01:20:20.710 align:middle line:90% That's actually one power of-- 01:20:20.710 --> 01:20:23.120 align:middle line:84% AUDIENCE: I mean, if you want it to actually succeed-- 01:20:23.120 --> 01:20:24.550 align:middle line:84% RAMESH RASKAR: But when you reconstruct, 01:20:24.550 --> 01:20:26.342 align:middle line:84% you have to know something about the scene. 01:20:26.342 --> 01:20:28.800 align:middle line:84% You have to know in which transform basis is actually 01:20:28.800 --> 01:20:29.300 align:middle line:90% sparse. 01:20:29.300 --> 01:20:31.810 align:middle line:84% So is it sparse when you take a Fourier transform? 01:20:31.810 --> 01:20:33.935 align:middle line:84% Is it sparse when you take a [INAUDIBLE] transform? 01:20:33.935 --> 01:20:35.950 align:middle line:84% Is it sparse when you take gradients? 01:20:35.950 --> 01:20:37.930 align:middle line:84% Like in terms of cartoons, it's just gradients. 01:20:37.930 --> 01:20:40.450 align:middle line:84% So you have to know that when you do reconstruction. 01:20:40.450 --> 01:20:43.240 align:middle line:84% But advantages are, at the time of capture, 01:20:43.240 --> 01:20:45.850 align:middle line:84% I just use this random basis or Fourier basis-- 01:20:45.850 --> 01:20:48.430 align:middle line:84% I mean, kind of a modified [INAUDIBLE] basis. 01:20:48.430 --> 01:20:50.110 align:middle line:84% I can just go ahead and spec sample it. 01:20:50.110 --> 01:20:51.730 align:middle line:84% And in software and reconstruction, 01:20:51.730 --> 01:20:55.300 align:middle line:84% I don't worry about some prior information about the scene, 01:20:55.300 --> 01:20:56.890 align:middle line:90% which is great. 01:20:56.890 --> 01:21:00.280 align:middle line:84% AUDIENCE: Well, I think in the case where you're just 01:21:00.280 --> 01:21:06.430 align:middle line:84% taking a set style of captures, that you're limiting yourself 01:21:06.430 --> 01:21:09.500 align:middle line:84% in what kind of scenes will be compatible with that capture. 01:21:09.500 --> 01:21:12.490 align:middle line:84% So for example, if I just had a scene that's all white, 01:21:12.490 --> 01:21:14.300 align:middle line:84% then just one captured would be enough. 01:21:14.300 --> 01:21:15.542 align:middle line:84% RAMESH RASKAR: Yeah, but that's because you 01:21:15.542 --> 01:21:16.240 align:middle line:90% know something about the scene. 01:21:16.240 --> 01:21:16.870 align:middle line:90% AUDIENCE: Exactly. 01:21:16.870 --> 01:21:17.370 align:middle line:90% Exactly. 01:21:17.370 --> 01:21:18.010 align:middle line:90% So-- 01:21:18.010 --> 01:21:20.453 align:middle line:84% RAMESH RASKAR: But if you have this situation where 01:21:20.453 --> 01:21:22.120 align:middle line:84% you don't know anything about the scene, 01:21:22.120 --> 01:21:25.630 align:middle line:84% you'll just use the same exact procedure, for example. 01:21:25.630 --> 01:21:27.130 align:middle line:84% AUDIENCE: Right, but you don't, then 01:21:27.130 --> 01:21:30.100 align:middle line:84% you lose the benefit of taking less pictures. 01:21:30.100 --> 01:21:31.900 align:middle line:84% RAMESH RASKAR: No, the claim is that even 01:21:31.900 --> 01:21:33.692 align:middle line:84% if you don't know anything about the scene, 01:21:33.692 --> 01:21:35.470 align:middle line:90% you take very few measurements. 01:21:35.470 --> 01:21:38.020 align:middle line:84% All you know about the scene is that once you 01:21:38.020 --> 01:21:42.100 align:middle line:84% take its transform, some transform, it's very sparse. 01:21:42.100 --> 01:21:44.253 align:middle line:84% It can be represented in a complex place. 01:21:44.253 --> 01:21:45.670 align:middle line:84% AUDIENCE: So I remember, actually, 01:21:45.670 --> 01:21:47.560 align:middle line:84% a mathematical mapping for this, where 01:21:47.560 --> 01:21:51.250 align:middle line:84% we're reducing dynamically the number of captures 01:21:51.250 --> 01:21:53.170 align:middle line:84% you have to take while you're capturing it. 01:21:53.170 --> 01:21:54.550 align:middle line:84% RAMESH RASKAR: But that's adaptive measure-- adaptive 01:21:54.550 --> 01:21:54.760 align:middle line:90% measure. 01:21:54.760 --> 01:21:55.000 align:middle line:90% AUDIENCE: Yes. 01:21:55.000 --> 01:21:56.620 align:middle line:84% RAMESH RASKAR: Because once you take a picture, you say, 01:21:56.620 --> 01:21:58.550 align:middle line:84% let me see what I did not capture. 01:21:58.550 --> 01:22:00.280 align:middle line:84% So let me take the next one next. 01:22:00.280 --> 01:22:02.460 align:middle line:90% That's a very different problem. 01:22:02.460 --> 01:22:03.085 align:middle line:90% AUDIENCE: Yeah. 01:22:03.085 --> 01:22:05.585 align:middle line:84% Well, anyways, the code for it's inside the dual photography 01:22:05.585 --> 01:22:06.087 align:middle line:90% thing. 01:22:06.087 --> 01:22:08.170 align:middle line:84% RAMESH RASKAR: Yeah, somebody did dual photography 01:22:08.170 --> 01:22:09.040 align:middle line:90% with compressive sensing. 01:22:09.040 --> 01:22:09.700 align:middle line:90% AUDIENCE: And it's adaptive. 01:22:09.700 --> 01:22:11.050 align:middle line:84% RAMESH RASKAR: And it works very well because, again, it's 01:22:11.050 --> 01:22:13.817 align:middle line:84% a high-dimensional signal, 2D camera, 2D projector. 01:22:13.817 --> 01:22:16.150 align:middle line:84% It's four dimensional, but what you're trying to recover 01:22:16.150 --> 01:22:17.860 align:middle line:90% is two dimensional. 01:22:17.860 --> 01:22:18.680 align:middle line:90% So it works again. 01:22:18.680 --> 01:22:19.763 align:middle line:90% So tomography is the same. 01:22:19.763 --> 01:22:24.130 align:middle line:84% It's 4D capture for 3D representation. 01:22:24.130 --> 01:22:27.970 align:middle line:84% OK, so I'm sorry we're not taking a break. 01:22:27.970 --> 01:22:31.540 align:middle line:84% Should we take a 30-second break before we 01:22:31.540 --> 01:22:34.630 align:middle line:84% move on to two very small topics, which 01:22:34.630 --> 01:22:39.690 align:middle line:84% is how to write a paper and wishlist for photography.