WEBVTT 00:00:00.000 --> 00:00:02.490 align:middle line:84% The following content is provided under a Creative 00:00:02.490 --> 00:00:03.730 align:middle line:90% Commons license. 00:00:03.730 --> 00:00:06.120 align:middle line:84% Your support will help MIT OpenCourseWare 00:00:06.120 --> 00:00:10.090 align:middle line:84% continue to offer high quality educational resources for free. 00:00:10.090 --> 00:00:12.720 align:middle line:84% To make a donation or to view additional materials 00:00:12.720 --> 00:00:16.650 align:middle line:84% from hundreds of MIT courses, visit MIT OpenCourseWare 00:00:16.650 --> 00:00:17.910 align:middle line:90% at ocw.mit.edu. 00:00:17.910 --> 00:00:21.460 align:middle line:90% 00:00:21.460 --> 00:00:24.370 align:middle line:84% RAMESH RASKAR: The final project is a very critical part 00:00:24.370 --> 00:00:26.090 align:middle line:90% of this class. 00:00:26.090 --> 00:00:27.970 align:middle line:90% So I'll emphasize this. 00:00:27.970 --> 00:00:30.910 align:middle line:84% The assignments are simple and straightforward. 00:00:30.910 --> 00:00:33.100 align:middle line:84% If you're struggling with it, talk to me, 00:00:33.100 --> 00:00:35.620 align:middle line:84% or [? Ankit, ?] or Professor [INAUDIBLE],, 00:00:35.620 --> 00:00:40.100 align:middle line:84% or Professor [? Oliveira. ?] And we are here to help you. 00:00:40.100 --> 00:00:43.300 align:middle line:84% But the final project is really, really critical. 00:00:43.300 --> 00:00:47.440 align:middle line:90% 00:00:47.440 --> 00:00:49.930 align:middle line:90% The project has to be novel. 00:00:49.930 --> 00:00:53.370 align:middle line:84% It has to be something nobody has done before. 00:00:53.370 --> 00:00:56.810 align:middle line:84% Or at least, nobody's done it the way you are doing it. 00:00:56.810 --> 00:00:59.770 align:middle line:84% And so the problem statement has to be novel. 00:00:59.770 --> 00:01:03.340 align:middle line:84% Its execution has to be beautiful. 00:01:03.340 --> 00:01:07.450 align:middle line:84% And its impact-- you should reap some results. 00:01:07.450 --> 00:01:09.850 align:middle line:84% You should show that you are solving it 00:01:09.850 --> 00:01:13.180 align:middle line:84% or it's possible to evaluate what you have done. 00:01:13.180 --> 00:01:16.240 align:middle line:90% 00:01:16.240 --> 00:01:21.520 align:middle line:84% And we have multiple stages to get you prepared 00:01:21.520 --> 00:01:24.160 align:middle line:90% for this final project. 00:01:24.160 --> 00:01:27.040 align:middle line:84% You have to come with at least three ideas 00:01:27.040 --> 00:01:29.620 align:middle line:90% for a final project. 00:01:29.620 --> 00:01:30.850 align:middle line:90% And this is very critical. 00:01:30.850 --> 00:01:35.050 align:middle line:84% If you go to the wiki on the camera culture group page, 00:01:35.050 --> 00:01:37.960 align:middle line:84% we have a whole section on how to come up with great ideas 00:01:37.960 --> 00:01:40.310 align:middle line:90% and how to brainstorm and so on. 00:01:40.310 --> 00:01:42.340 align:middle line:90% So I encourage you to do that. 00:01:42.340 --> 00:01:44.080 align:middle line:84% On our [? Stellar ?] page we also 00:01:44.080 --> 00:01:48.680 align:middle line:84% have a whole presentation on how to come up with new ideas. 00:01:48.680 --> 00:01:53.980 align:middle line:84% There are six ways of coming up with new ideas. 00:01:53.980 --> 00:01:56.380 align:middle line:84% And you should start setting up a meeting with me 00:01:56.380 --> 00:02:01.228 align:middle line:84% or any one of these people in the next week. 00:02:01.228 --> 00:02:02.770 align:middle line:84% If you're in the building, it's easy. 00:02:02.770 --> 00:02:04.810 align:middle line:90% You can just catch me. 00:02:04.810 --> 00:02:07.240 align:middle line:84% Usually, between 5:00 PM to 6:00 PM 00:02:07.240 --> 00:02:10.210 align:middle line:84% is the best because there are no officially scheduled meetings. 00:02:10.210 --> 00:02:13.360 align:middle line:84% You can just come to my office, and we can chat. 00:02:13.360 --> 00:02:16.120 align:middle line:84% If you're not taking this class for credit 00:02:16.120 --> 00:02:18.640 align:middle line:84% and you're a listener, I would really 00:02:18.640 --> 00:02:23.900 align:middle line:84% appreciate if you will pitch an idea for a project. 00:02:23.900 --> 00:02:28.510 align:middle line:84% And we'll have an opportunity to do that on October 30. 00:02:28.510 --> 00:02:31.630 align:middle line:84% So just come up here and say, I have this crazy idea. 00:02:31.630 --> 00:02:38.650 align:middle line:84% If somebody wants to work on this, let's team up. 00:02:38.650 --> 00:02:45.400 align:middle line:84% And then I want you to send a very simple email with-- 00:02:45.400 --> 00:02:49.060 align:middle line:84% after discussing these three ideas, I, or [? Ankit, ?] 00:02:49.060 --> 00:02:51.700 align:middle line:84% or our other two mentors will help you narrow down 00:02:51.700 --> 00:02:55.210 align:middle line:84% to maybe the top two ideas, or maybe the top one idea. 00:02:55.210 --> 00:02:58.825 align:middle line:84% And then, for that, you need to send these five things. 00:02:58.825 --> 00:03:01.340 align:middle line:90% All right? 00:03:01.340 --> 00:03:05.190 align:middle line:84% And on November 6, we will do a three-minute presentation 00:03:05.190 --> 00:03:07.190 align:middle line:84% so everybody knows, in the class, 00:03:07.190 --> 00:03:09.110 align:middle line:84% the kind of problem you are attacking. 00:03:09.110 --> 00:03:11.660 align:middle line:84% Maybe there's some synergy between multiple projects. 00:03:11.660 --> 00:03:15.650 align:middle line:84% You can help each other find software, or talk to people, 00:03:15.650 --> 00:03:18.020 align:middle line:90% or get some equipment. 00:03:18.020 --> 00:03:21.170 align:middle line:84% And then final proposals are due two weeks 00:03:21.170 --> 00:03:25.890 align:middle line:84% before the actual final presentation. 00:03:25.890 --> 00:03:28.340 align:middle line:84% And by then you should have some initial experiments. 00:03:28.340 --> 00:03:32.120 align:middle line:84% Remember, your final assignment is due on November 13. 00:03:32.120 --> 00:03:34.580 align:middle line:84% So you can't start thinking about your final project 00:03:34.580 --> 00:03:36.170 align:middle line:84% after the final assignment is due. 00:03:36.170 --> 00:03:37.640 align:middle line:90% You need to start thinking now. 00:03:37.640 --> 00:03:40.593 align:middle line:90% 00:03:40.593 --> 00:03:42.010 align:middle line:84% And at this stage, it doesn't have 00:03:42.010 --> 00:03:43.750 align:middle line:90% to be completely hashed out. 00:03:43.750 --> 00:03:45.213 align:middle line:90% That's why you're in the class. 00:03:45.213 --> 00:03:46.630 align:middle line:84% We'll help you think through that. 00:03:46.630 --> 00:03:49.700 align:middle line:84% And then December 4, the class is finished. 00:03:49.700 --> 00:03:54.680 align:middle line:84% You finish really early, because as I said, we meet on Fridays. 00:03:54.680 --> 00:03:58.750 align:middle line:84% And the Thursday after this, so December 10, 00:03:58.750 --> 00:04:00.325 align:middle line:90% is the last day of classes. 00:04:00.325 --> 00:04:02.200 align:middle line:84% So we are one of the first classes to finish. 00:04:02.200 --> 00:04:04.840 align:middle line:90% 00:04:04.840 --> 00:04:09.790 align:middle line:84% Unfortunately, November 27 is a Thanksgiving break. 00:04:09.790 --> 00:04:12.790 align:middle line:84% And so-- which means we don't have a class 00:04:12.790 --> 00:04:16.399 align:middle line:84% just before our final projects to discuss and do other things. 00:04:16.399 --> 00:04:19.149 align:middle line:84% But I will be available throughout that time. 00:04:19.149 --> 00:04:21.203 align:middle line:90% And we can help you. 00:04:21.203 --> 00:04:22.870 align:middle line:84% If you need equipment, you need cameras, 00:04:22.870 --> 00:04:28.300 align:middle line:84% you need software, whatever-- we can try to help you. 00:04:28.300 --> 00:04:30.160 align:middle line:90% Any questions on final projects? 00:04:30.160 --> 00:04:36.240 align:middle line:90% 00:04:36.240 --> 00:04:37.980 align:middle line:84% And Mike was saying that it might 00:04:37.980 --> 00:04:44.290 align:middle line:84% be good to know what students did last year in-- 00:04:44.290 --> 00:04:47.610 align:middle line:84% as last projects, and how they decided, and so on. 00:04:47.610 --> 00:04:49.560 align:middle line:84% So Mike Hirsch is going to come and talk today 00:04:49.560 --> 00:04:51.030 align:middle line:90% about his [? pilot ?] screen. 00:04:51.030 --> 00:04:52.710 align:middle line:84% And that would be a good way to ask him, 00:04:52.710 --> 00:04:55.753 align:middle line:84% how did he get started, and how did they consider the paper? 00:04:55.753 --> 00:04:59.610 align:middle line:90% 00:04:59.610 --> 00:05:02.130 align:middle line:84% So today we're going to talk about lightfields-- just finish 00:05:02.130 --> 00:05:07.260 align:middle line:84% up the lightfields, and then talk about cameras for SCI. 00:05:07.260 --> 00:05:13.410 align:middle line:84% So we said there are basically three ways of coming-- 00:05:13.410 --> 00:05:15.660 align:middle line:90% of capturing a lightfield. 00:05:15.660 --> 00:05:21.300 align:middle line:84% Can somebody tell me in two sentences what's a lightfield? 00:05:21.300 --> 00:05:22.550 align:middle line:90% AUDIENCE: On the real cameras? 00:05:22.550 --> 00:05:25.100 align:middle line:84% RAMESH RASKAR: In a real-- inside a camera? 00:05:25.100 --> 00:05:25.640 align:middle line:90% What is it? 00:05:25.640 --> 00:05:28.460 align:middle line:90% 00:05:28.460 --> 00:05:32.220 align:middle line:84% How are they represented mathematically? 00:05:32.220 --> 00:05:36.120 align:middle line:84% A representation-- just a representation. 00:05:36.120 --> 00:05:38.390 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:05:38.390 --> 00:05:39.390 align:middle line:90% RAMESH RASKAR: So it's-- 00:05:39.390 --> 00:05:40.440 align:middle line:84% how many dimensions does it have [INAUDIBLE]?? 00:05:40.440 --> 00:05:41.180 align:middle line:90% AUDIENCE: Four dimensions. 00:05:41.180 --> 00:05:41.550 align:middle line:90% Four. 00:05:41.550 --> 00:05:43.092 align:middle line:84% RAMESH RASKAR: It's four-dimensional. 00:05:43.092 --> 00:05:45.187 align:middle line:84% Although the world seems to be two-dimensional, 00:05:45.187 --> 00:05:46.770 align:middle line:84% [INAUDIBLE] actually four-dimensional. 00:05:46.770 --> 00:05:50.149 align:middle line:84% And where do we get those four dimensions from? 00:05:50.149 --> 00:05:51.130 align:middle line:90% We have a lens. 00:05:51.130 --> 00:05:54.715 align:middle line:90% 00:05:54.715 --> 00:05:56.560 align:middle line:90% And we got a sensor. 00:05:56.560 --> 00:06:00.460 align:middle line:84% So in flatland it's just two-dimensional [INAUDIBLE] 00:06:00.460 --> 00:06:01.002 align:middle line:90% wall. 00:06:01.002 --> 00:06:02.710 align:middle line:84% Where are the two dimensions coming from? 00:06:02.710 --> 00:06:05.840 align:middle line:90% Well, the x dimension is easy. 00:06:05.840 --> 00:06:07.030 align:middle line:90% That's one dimension. 00:06:07.030 --> 00:06:08.290 align:middle line:90% And the other dimension is? 00:06:08.290 --> 00:06:09.540 align:middle line:90% AUDIENCE: Through [INAUDIBLE]. 00:06:09.540 --> 00:06:12.670 align:middle line:84% RAMESH RASKAR: [INAUDIBLE],, That's this dimension. 00:06:12.670 --> 00:06:15.570 align:middle line:84% So if you connect any two points, 00:06:15.570 --> 00:06:18.150 align:middle line:84% that indicates the direction of the rate. 00:06:18.150 --> 00:06:22.680 align:middle line:84% And then flatline, [INAUDIBLE],, all the raised bits 00:06:22.680 --> 00:06:24.270 align:middle line:90% is two dimensional. 00:06:24.270 --> 00:06:28.390 align:middle line:84% In the real world [INAUDIBLE] it's four dimensional. 00:06:28.390 --> 00:06:33.710 align:middle line:84% Because your sensor is going to be X and Y. 00:06:33.710 --> 00:06:45.950 align:middle line:84% And your lens is going to have [? VW. ?] 00:06:45.950 --> 00:06:47.840 align:middle line:84% So that's the lightfield that we care about. 00:06:47.840 --> 00:06:51.190 align:middle line:84% And why do we care about lightfields so much? 00:06:51.190 --> 00:06:54.810 align:middle line:84% What's so unique about lightfield? 00:06:54.810 --> 00:06:57.375 align:middle line:90% 00:06:57.375 --> 00:06:59.000 align:middle line:84% AUDIENCE: If we capture the lightfield, 00:06:59.000 --> 00:07:01.452 align:middle line:84% we have captured almost everything that we can capture. 00:07:01.452 --> 00:07:02.410 align:middle line:90% RAMESH RASKAR: Exactly. 00:07:02.410 --> 00:07:05.468 align:middle line:84% It's a complete representation of light 00:07:05.468 --> 00:07:06.510 align:middle line:90% that's entering the lens. 00:07:06.510 --> 00:07:11.780 align:middle line:84% So anything that you could ever imagine doing with dimension-- 00:07:11.780 --> 00:07:16.670 align:middle line:84% changing focus, changing zoom, changing aperture size-- 00:07:16.670 --> 00:07:19.430 align:middle line:84% all those things are already captured 00:07:19.430 --> 00:07:22.680 align:middle line:90% in this four [INAUDIBLE]. 00:07:22.680 --> 00:07:24.890 align:middle line:84% And from there you can do anything you want. 00:07:24.890 --> 00:07:28.080 align:middle line:84% And if you cannot do it using mechanically changing camera 00:07:28.080 --> 00:07:31.280 align:middle line:84% parameters, you cannot do it with lightfields. 00:07:31.280 --> 00:07:33.140 align:middle line:84% Sorry-- if you cannot do it with lightfield, 00:07:33.140 --> 00:07:37.499 align:middle line:84% you cannot do it by changing [INAUDIBLE].. 00:07:37.499 --> 00:07:40.560 align:middle line:84% So it's a very, very powerful way of-- 00:07:40.560 --> 00:07:43.020 align:middle line:84% and as we saw earlier, the lightfield 00:07:43.020 --> 00:07:45.855 align:middle line:84% also represents the [? waveguard, ?] both phase 00:07:45.855 --> 00:07:50.180 align:middle line:90% and function of [INAUDIBLE]. 00:07:50.180 --> 00:07:52.322 align:middle line:84% So given that this is so important, 00:07:52.322 --> 00:07:54.780 align:middle line:84% photographers never thought about capturing the lightfield. 00:07:54.780 --> 00:07:57.580 align:middle line:90% [INAUDIBLE] capturing light. 00:07:57.580 --> 00:07:59.260 align:middle line:84% And that's why this is computational-- 00:07:59.260 --> 00:08:00.972 align:middle line:84% from a pure computational camera, 00:08:00.972 --> 00:08:03.180 align:middle line:84% because we're trying to understand this relationship. 00:08:03.180 --> 00:08:05.872 align:middle line:84% And this is what allows us to build cool camera toys, 00:08:05.872 --> 00:08:07.330 align:middle line:84% and also come up with [INAUDIBLE].. 00:08:07.330 --> 00:08:10.420 align:middle line:90% 00:08:10.420 --> 00:08:14.680 align:middle line:84% So given that, what are the three ways we can capture this? 00:08:14.680 --> 00:08:20.800 align:middle line:90% 00:08:20.800 --> 00:08:24.280 align:middle line:84% One is using a lens [INAUDIBLE] that we 00:08:24.280 --> 00:08:27.886 align:middle line:90% saw earlier [INAUDIBLE]. 00:08:27.886 --> 00:08:31.120 align:middle line:84% All you want to do is, right now, 00:08:31.120 --> 00:08:34.669 align:middle line:84% at this pixel, a ray from here and a ray from here. 00:08:34.669 --> 00:08:35.530 align:middle line:90% They converge. 00:08:35.530 --> 00:08:38.903 align:middle line:84% And any variation in the radius along those rays is lost. 00:08:38.903 --> 00:08:40.570 align:middle line:84% We want to make sure that we can capture 00:08:40.570 --> 00:08:43.909 align:middle line:90% this and this individually. 00:08:43.909 --> 00:08:45.453 align:middle line:84% So what are some ways we can do it? 00:08:45.453 --> 00:08:46.870 align:middle line:84% Instead of putting the sensor here 00:08:46.870 --> 00:08:49.930 align:middle line:84% we can put the sensor further back, and then [INAUDIBLE].. 00:08:49.930 --> 00:08:53.540 align:middle line:90% 00:08:53.540 --> 00:08:56.720 align:middle line:84% And the light can [INAUDIBLE] here [INAUDIBLE] 00:08:56.720 --> 00:08:59.975 align:middle line:90% for the lighting. 00:08:59.975 --> 00:09:02.710 align:middle line:90% [? Inputs ?] here. 00:09:02.710 --> 00:09:06.110 align:middle line:84% And it'll be mapped [INAUDIBLE] different pixels. 00:09:06.110 --> 00:09:09.540 align:middle line:84% So you have [INAUDIBLE] equals 0, 00:09:09.540 --> 00:09:16.640 align:middle line:84% [INAUDIBLE] equals this [INAUDIBLE] and plus 4, 00:09:16.640 --> 00:09:19.310 align:middle line:90% and [INAUDIBLE] minus 4. 00:09:19.310 --> 00:09:23.030 align:middle line:84% Then have captured-- for this given x, 00:09:23.030 --> 00:09:28.160 align:middle line:84% I've captured-- let's say this is 1, 2, 3, 4, 5. 00:09:28.160 --> 00:09:30.470 align:middle line:84% Then have captured their 3 comma-- 00:09:30.470 --> 00:09:32.210 align:middle line:90% well, minus 4. 00:09:32.210 --> 00:09:34.010 align:middle line:90% AUDIENCE: Minus 4. 00:09:34.010 --> 00:09:37.812 align:middle line:84% RAMESH RASKAR: And yet I've captured 3 comma [INAUDIBLE].. 00:09:37.812 --> 00:09:40.870 align:middle line:90% 00:09:40.870 --> 00:09:42.592 align:middle line:84% So I have captured all [INAUDIBLE].. 00:09:42.592 --> 00:09:44.050 align:middle line:84% But of course, you won't have what? 00:09:44.050 --> 00:09:45.610 align:middle line:90% AUDIENCE: Resolution. 00:09:45.610 --> 00:09:47.318 align:middle line:84% RAMESH RASKAR: You won't have resolution. 00:09:47.318 --> 00:09:52.510 align:middle line:84% Because for x equals 3.5 they're just opaque. 00:09:52.510 --> 00:09:54.750 align:middle line:90% So that light is lost. 00:09:54.750 --> 00:09:56.680 align:middle line:90% [INAUDIBLE] captured. 00:09:56.680 --> 00:10:00.670 align:middle line:90% So if I say here-- 00:10:00.670 --> 00:10:02.290 align:middle line:90% let's say I have 1,000 pixels. 00:10:02.290 --> 00:10:08.990 align:middle line:84% So that's 900 pixels [INAUDIBLE].. 00:10:08.990 --> 00:10:14.270 align:middle line:84% And I'm going to chop it into [INAUDIBLE] plus 4, minus 4. 00:10:14.270 --> 00:10:15.920 align:middle line:90% What is the final-- 00:10:15.920 --> 00:10:17.360 align:middle line:84% how many pinholes can I put here? 00:10:17.360 --> 00:10:21.990 align:middle line:90% 00:10:21.990 --> 00:10:33.010 align:middle line:84% So this total resolution is x resolution times [INAUDIBLE].. 00:10:33.010 --> 00:10:38.910 align:middle line:84% In a traditional camera, 900 is just your execution. 00:10:38.910 --> 00:10:41.550 align:middle line:84% But now we're going to also try to capture the [INAUDIBLE] 00:10:41.550 --> 00:10:45.600 align:middle line:84% variations in the same number of [INAUDIBLE].. 00:10:45.600 --> 00:10:49.610 align:middle line:84% So we know that our [INAUDIBLE] resolution is 9. 00:10:49.610 --> 00:10:53.013 align:middle line:90% So [INAUDIBLE] 100. 00:10:53.013 --> 00:10:55.430 align:middle line:84% Which means that our image is only going to be 100 pixels. 00:10:55.430 --> 00:10:56.930 align:middle line:90% [INAUDIBLE] only 100 pinholes. 00:10:56.930 --> 00:11:02.610 align:middle line:90% 00:11:02.610 --> 00:11:08.210 align:middle line:84% And for each pinhole I'm going to get the image [INAUDIBLE] 00:11:08.210 --> 00:11:08.710 align:middle line:90% pixels. 00:11:08.710 --> 00:11:12.450 align:middle line:90% 00:11:12.450 --> 00:11:16.110 align:middle line:84% And from this 100 times 9 image, which is 900 pixels, 00:11:16.110 --> 00:11:20.400 align:middle line:84% I can create a lightfield where the spatial resolution is 100 00:11:20.400 --> 00:11:23.760 align:middle line:90% and the angular resolution is 9. 00:11:23.760 --> 00:11:27.360 align:middle line:90% And what's the disadvantage? 00:11:27.360 --> 00:11:28.920 align:middle line:90% One is that you lose resolution. 00:11:28.920 --> 00:11:30.278 align:middle line:90% [INTERPOSING VOICES] 00:11:30.278 --> 00:11:30.945 align:middle line:90% AUDIENCE: Light. 00:11:30.945 --> 00:11:35.280 align:middle line:84% RAMESH RASKAR: You lose light, because all the lights that's 00:11:35.280 --> 00:11:38.560 align:middle line:84% going through this opaque area is completely lost. 00:11:38.560 --> 00:11:41.010 align:middle line:84% It's like looking through multiple pinhole cameras. 00:11:41.010 --> 00:11:44.640 align:middle line:84% It's like looking at the world through [INAUDIBLE] holes. 00:11:44.640 --> 00:11:46.860 align:middle line:90% So most of the light is lost. 00:11:46.860 --> 00:11:50.910 align:middle line:84% In this case, let's say out of nine pixels, one pixel is open. 00:11:50.910 --> 00:11:55.950 align:middle line:84% Then only 1/9 of the light is being captured. 00:11:55.950 --> 00:11:59.010 align:middle line:84% The other 8/9 of the light is lost. 00:11:59.010 --> 00:12:02.130 align:middle line:90% So that's a big problem. 00:12:02.130 --> 00:12:07.990 align:middle line:84% But still, conceptually, this is very key, because you can say, 00:12:07.990 --> 00:12:11.250 align:middle line:90% I want to capture space-- 00:12:11.250 --> 00:12:14.580 align:middle line:84% space radiation as well as angle resolution-- angle radiation. 00:12:14.580 --> 00:12:16.800 align:middle line:84% So I'm just going to chop my [? word ?] 00:12:16.800 --> 00:12:23.684 align:middle line:90% into 100 pinholes and 9 angle-- 00:12:23.684 --> 00:12:25.980 align:middle line:90% [INAUDIBLE] angle spaces. 00:12:25.980 --> 00:12:28.110 align:middle line:84% And I'm going to capture nine pixels, and so on. 00:12:28.110 --> 00:12:30.130 align:middle line:90% It's a very clean, simple model. 00:12:30.130 --> 00:12:33.253 align:middle line:84% But in the real world, this is very inefficient. 00:12:33.253 --> 00:12:35.670 align:middle line:84% It's just like for a camera, if you're [INAUDIBLE] pinhole 00:12:35.670 --> 00:12:36.225 align:middle line:90% model-- 00:12:36.225 --> 00:12:37.260 align:middle line:90% pinhole camera model. 00:12:37.260 --> 00:12:39.240 align:middle line:84% But in the real world, [INAUDIBLE].. 00:12:39.240 --> 00:12:41.855 align:middle line:90% 00:12:41.855 --> 00:12:42.355 align:middle line:90% Now-- 00:12:42.355 --> 00:12:44.650 align:middle line:84% AUDIENCE: So let me ask you this. 00:12:44.650 --> 00:12:47.330 align:middle line:84% Can you really construct this in such a way that, 00:12:47.330 --> 00:12:48.850 align:middle line:90% as you go to the end of the-- 00:12:48.850 --> 00:12:50.980 align:middle line:90% I mean, your x. 00:12:50.980 --> 00:12:55.180 align:middle line:84% Aren't you wasting part of the nine spatial 00:12:55.180 --> 00:12:56.170 align:middle line:90% samples that you have? 00:12:56.170 --> 00:12:57.340 align:middle line:84% RAMESH RASKAR: You mean at the top or the bottom? 00:12:57.340 --> 00:12:57.970 align:middle line:90% AUDIENCE: Yeah. 00:12:57.970 --> 00:13:02.260 align:middle line:84% Because maybe you can't actually have this [INAUDIBLE] opening 00:13:02.260 --> 00:13:03.850 align:middle line:90% over the entire set. 00:13:03.850 --> 00:13:05.138 align:middle line:90% Yeah, exactly. 00:13:05.138 --> 00:13:06.430 align:middle line:90% RAMESH RASKAR: As long as you-- 00:13:06.430 --> 00:13:07.570 align:middle line:90% this is your sensor. 00:13:07.570 --> 00:13:10.660 align:middle line:84% And you can create a pinhole that can capture light 00:13:10.660 --> 00:13:12.580 align:middle line:90% from different directions. 00:13:12.580 --> 00:13:13.090 align:middle line:90% You're OK. 00:13:13.090 --> 00:13:15.307 align:middle line:90% There's not much you can do. 00:13:15.307 --> 00:13:17.140 align:middle line:84% I think you might-- what you may be thinking 00:13:17.140 --> 00:13:19.870 align:middle line:84% about is, this pinhole, at some angle, 00:13:19.870 --> 00:13:21.310 align:middle line:90% will become really opaque. 00:13:21.310 --> 00:13:22.300 align:middle line:84% You won't be able to see through it. 00:13:22.300 --> 00:13:22.870 align:middle line:90% Is that what you're saying? 00:13:22.870 --> 00:13:23.495 align:middle line:90% AUDIENCE: Yeah. 00:13:23.495 --> 00:13:26.410 align:middle line:84% What I'm saying is that maybe, through this particular angle, 00:13:26.410 --> 00:13:28.450 align:middle line:84% the pinhole doesn't span the whole nine 00:13:28.450 --> 00:13:30.022 align:middle line:90% pixels you have behind it. 00:13:30.022 --> 00:13:31.480 align:middle line:84% RAMESH RASKAR: That's a good point. 00:13:31.480 --> 00:13:33.620 align:middle line:84% But you are leading me to the next question, 00:13:33.620 --> 00:13:37.510 align:middle line:90% which is, this spacing here. 00:13:37.510 --> 00:13:39.310 align:middle line:84% What is magical about this spacing? 00:13:39.310 --> 00:13:41.410 align:middle line:84% Because if you don't have the right spacing, 00:13:41.410 --> 00:13:44.350 align:middle line:90% you'll get those problems. 00:13:44.350 --> 00:13:47.050 align:middle line:84% What needs to happen in the spacing so 00:13:47.050 --> 00:13:51.380 align:middle line:84% that you actually capture all these 900 [INAUDIBLE]?? 00:13:51.380 --> 00:13:51.880 align:middle line:90% Yes? 00:13:51.880 --> 00:13:53.255 align:middle line:84% AUDIENCE: [INAUDIBLE] they really 00:13:53.255 --> 00:13:55.585 align:middle line:90% should not overlap [INAUDIBLE]. 00:13:55.585 --> 00:13:57.010 align:middle line:90% RAMESH RASKAR: Exactly. 00:13:57.010 --> 00:13:57.700 align:middle line:90% Exactly. 00:13:57.700 --> 00:14:02.170 align:middle line:84% So if you think about the blob that's 00:14:02.170 --> 00:14:05.890 align:middle line:84% coming through from top to bottom, 00:14:05.890 --> 00:14:07.690 align:middle line:90% there'll be some loss here. 00:14:07.690 --> 00:14:14.660 align:middle line:84% If you see the blob that's coming through here, 00:14:14.660 --> 00:14:17.010 align:middle line:84% these blobs are just barely touching each other. 00:14:17.010 --> 00:14:23.630 align:middle line:84% So here we have theta equals plus 4, 3, 0, 1, minus 4. 00:14:23.630 --> 00:14:28.600 align:middle line:84% And then the next one is theta equals plus 4. 00:14:28.600 --> 00:14:30.340 align:middle line:90% So it's flipped [INAUDIBLE]. 00:14:30.340 --> 00:14:30.840 align:middle line:90% Yes. 00:14:30.840 --> 00:14:35.110 align:middle line:84% Then, from minus 44, you guys want to go from minus 44 00:14:35.110 --> 00:14:37.550 align:middle line:90% [INAUDIBLE] 00:14:37.550 --> 00:14:42.110 align:middle line:84% Now, if I move this pinhole further or back, 00:14:42.110 --> 00:14:44.330 align:middle line:84% you're going to get [INAUDIBLE] variables, right? 00:14:44.330 --> 00:14:48.148 align:middle line:84% Let's say I move this pinhole a little forward 00:14:48.148 --> 00:14:49.856 align:middle line:84% and I just take one pinhole [INAUDIBLE].. 00:14:49.856 --> 00:14:56.830 align:middle line:90% 00:14:56.830 --> 00:14:58.270 align:middle line:90% And everything else is the same. 00:14:58.270 --> 00:15:01.490 align:middle line:90% 00:15:01.490 --> 00:15:05.180 align:middle line:90% Are the blobs larger? 00:15:05.180 --> 00:15:06.710 align:middle line:84% If I place the pinhole at the same-- 00:15:06.710 --> 00:15:10.700 align:middle line:84% I place the pinhole exactly at 9 pixels away. 00:15:10.700 --> 00:15:14.030 align:middle line:84% If I can put it exactly 9 pixels away, 00:15:14.030 --> 00:15:19.680 align:middle line:84% the rays on the [? cloth ?] will slide into [INAUDIBLE].. 00:15:19.680 --> 00:15:23.856 align:middle line:84% So this blob here will start interfering with this blob here 00:15:23.856 --> 00:15:24.680 align:middle line:90% [INAUDIBLE]. 00:15:24.680 --> 00:15:28.467 align:middle line:90% 00:15:28.467 --> 00:15:29.300 align:middle line:90% So that's one issue. 00:15:29.300 --> 00:15:32.740 align:middle line:90% What happens if it's too close? 00:15:32.740 --> 00:15:35.065 align:middle line:84% Instead of moving further away, it gets too close? 00:15:35.065 --> 00:15:37.485 align:middle line:90% AUDIENCE: No spaces. 00:15:37.485 --> 00:15:39.870 align:middle line:84% RAMESH RASKAR: There will be a space between those blobs. 00:15:39.870 --> 00:15:46.270 align:middle line:84% Is that here [INAUDIBLE] a blob here and we get a blob here. 00:15:46.270 --> 00:15:50.180 align:middle line:90% And some pixels will be missing. 00:15:50.180 --> 00:15:52.730 align:middle line:84% So this is a very important point. 00:15:52.730 --> 00:15:54.950 align:middle line:84% And this is related to the numerical aperture, 00:15:54.950 --> 00:15:57.800 align:middle line:90% or the F-stop of the lens. 00:15:57.800 --> 00:16:00.260 align:middle line:90% So an F-stop of the lens-- 00:16:00.260 --> 00:16:01.940 align:middle line:84% again, pardon all this terminology. 00:16:01.940 --> 00:16:03.270 align:middle line:90% I don't like it. 00:16:03.270 --> 00:16:07.880 align:middle line:84% But [INAUDIBLE] about it is the f-number 00:16:07.880 --> 00:16:11.900 align:middle line:84% is simply the ratio of the diameter 00:16:11.900 --> 00:16:17.810 align:middle line:84% of the lens with the distance to the sensor. 00:16:17.810 --> 00:16:18.635 align:middle line:90% So it's a diameter. 00:16:18.635 --> 00:16:22.490 align:middle line:90% 00:16:22.490 --> 00:16:29.750 align:middle line:84% And your distance of the [INAUDIBLE].. 00:16:29.750 --> 00:16:36.580 align:middle line:84% The relationship between the two is just c over [INAUDIBLE] 00:16:36.580 --> 00:16:38.110 align:middle line:90% so if you have a-- 00:16:38.110 --> 00:16:42.140 align:middle line:84% and remember, the diameter is in the denominator, 00:16:42.140 --> 00:16:46.410 align:middle line:84% which means that as you have a larger lens, 00:16:46.410 --> 00:16:53.310 align:middle line:90% the f-number actually goes down. 00:16:53.310 --> 00:16:54.720 align:middle line:90% [INAUDIBLE] 00:16:54.720 --> 00:16:58.680 align:middle line:84% So let's take a concrete example. 00:16:58.680 --> 00:17:05.940 align:middle line:84% Let's say my lens has a diameter of 25 millimeters. 00:17:05.940 --> 00:17:08.250 align:middle line:84% And my focal length and distance is 00:17:08.250 --> 00:17:15.280 align:middle line:84% about [INAUDIBLE] millimeters, then the f-number is what? 00:17:15.280 --> 00:17:16.748 align:middle line:90% 50 [INAUDIBLE]. 00:17:16.748 --> 00:17:24.220 align:middle line:90% 00:17:24.220 --> 00:17:26.740 align:middle line:84% Does anybody know what the f-number of [INAUDIBLE]?? 00:17:26.740 --> 00:17:32.551 align:middle line:90% 00:17:32.551 --> 00:17:37.628 align:middle line:90% [INAUDIBLE] 0218. 00:17:37.628 --> 00:17:42.080 align:middle line:90% Then there is 4 and so on. 00:17:42.080 --> 00:17:48.250 align:middle line:84% So as you can see that if I make my diameter half-- 00:17:48.250 --> 00:17:51.270 align:middle line:90% 00:17:51.270 --> 00:17:57.260 align:middle line:84% so instead of 25 millimeter, I have 12.5 millimeter-- 00:17:57.260 --> 00:17:58.260 align:middle line:90% then what do I get here? 00:17:58.260 --> 00:18:02.150 align:middle line:90% I get 50 divided by 12.5. 00:18:02.150 --> 00:18:07.620 align:middle line:84% So that's [INAUDIBLE] the f-number of a 12.5 millimeter 00:18:07.620 --> 00:18:10.570 align:middle line:84% lens with a 50 millimeter focal length is 4. 00:18:10.570 --> 00:18:13.600 align:middle line:90% 00:18:13.600 --> 00:18:17.410 align:middle line:90% From 2 we jump to 4. 00:18:17.410 --> 00:18:20.632 align:middle line:84% What should we do to jump from 2 to 2.8? 00:18:20.632 --> 00:18:24.340 align:middle line:90% This is where the [INAUDIBLE]. 00:18:24.340 --> 00:18:27.150 align:middle line:84% But maybe some of you have forgotten [INAUDIBLE].. 00:18:27.150 --> 00:18:27.650 align:middle line:90% Yes? 00:18:27.650 --> 00:18:29.195 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:18:29.195 --> 00:18:30.320 align:middle line:90% RAMESH RASKAR: [INAUDIBLE]. 00:18:30.320 --> 00:18:33.340 align:middle line:90% 00:18:33.340 --> 00:18:36.330 align:middle line:84% How do you go from 2 to 2.0 [INAUDIBLE].. 00:18:36.330 --> 00:18:37.987 align:middle line:90% But how do you get [? 4.4? ?] 00:18:37.987 --> 00:18:40.940 align:middle line:90% AUDIENCE: [INAUDIBLE]. 00:18:40.940 --> 00:18:44.588 align:middle line:84% RAMESH RASKAR: So what's [INAUDIBLE]?? 00:18:44.588 --> 00:18:47.030 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:18:47.030 --> 00:18:52.910 align:middle line:84% RAMESH RASKAR: 1.41428 something, right? 00:18:52.910 --> 00:18:54.470 align:middle line:84% As far as [INAUDIBLE] are concerned, 00:18:54.470 --> 00:19:02.470 align:middle line:84% it's [INAUDIBLE] because we like to think about this size 00:19:02.470 --> 00:19:03.610 align:middle line:90% numbers. 00:19:03.610 --> 00:19:05.713 align:middle line:84% And this problem [INAUDIBLE] even bigger problem 00:19:05.713 --> 00:19:06.880 align:middle line:90% once you go for [INAUDIBLE]. 00:19:06.880 --> 00:19:10.504 align:middle line:84% Anyway, so why-- where are we getting this [INAUDIBLE]?? 00:19:10.504 --> 00:19:13.410 align:middle line:90% 00:19:13.410 --> 00:19:18.510 align:middle line:84% If you go from [INAUDIBLE] when you go from 25 to 12.5, 00:19:18.510 --> 00:19:21.840 align:middle line:84% the amount of light that's collected by this [INAUDIBLE] 00:19:21.840 --> 00:19:24.099 align:middle line:84% across this distance is related by what factor? 00:19:24.099 --> 00:19:24.974 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:19:24.974 --> 00:19:26.565 align:middle line:90% RAMESH RASKAR: 5 times 4. 00:19:26.565 --> 00:19:29.010 align:middle line:84% So the diameter is [INAUDIBLE] plus 2, 00:19:29.010 --> 00:19:33.870 align:middle line:84% but the area is decreasing by a factor of 4. 00:19:33.870 --> 00:19:38.050 align:middle line:84% Now, if you want-- so four times [INAUDIBLE] coming in. 00:19:38.050 --> 00:19:42.390 align:middle line:84% On the other hand, if you want to go from 25 to some other 00:19:42.390 --> 00:19:47.760 align:middle line:84% diameter so that you get half the width [INAUDIBLE]---- 00:19:47.760 --> 00:19:51.910 align:middle line:84% if you divide 25 [INAUDIBLE],, whatever that is-- 00:19:51.910 --> 00:19:57.862 align:middle line:90% 00:19:57.862 --> 00:20:00.480 align:middle line:90% 18, something like that? 00:20:00.480 --> 00:20:02.370 align:middle line:90% 25 [INAUDIBLE]. 00:20:02.370 --> 00:20:04.100 align:middle line:84% Let's say it's 18 point something. 00:20:04.100 --> 00:20:05.006 align:middle line:90% That fair? 00:20:05.006 --> 00:20:07.724 align:middle line:90% 00:20:07.724 --> 00:20:09.020 align:middle line:90% [INAUDIBLE] 00:20:09.020 --> 00:20:11.360 align:middle line:90% I'm being imprecise. 00:20:11.360 --> 00:20:12.290 align:middle line:90% [INAUDIBLE] 00:20:12.290 --> 00:20:15.208 align:middle line:90% 00:20:15.208 --> 00:20:17.750 align:middle line:84% So when I go from here to here, [INAUDIBLE] of 2 in diameter, 00:20:17.750 --> 00:20:18.940 align:middle line:90% I get half the length. 00:20:18.940 --> 00:20:21.890 align:middle line:84% And that's what photographers mean to go by-- 00:20:21.890 --> 00:20:24.620 align:middle line:90% go down by 1 F-stop. 00:20:24.620 --> 00:20:27.380 align:middle line:84% So when you go down by one F-stop, 00:20:27.380 --> 00:20:29.690 align:middle line:90% are you going from 2 to 3? 00:20:29.690 --> 00:20:32.930 align:middle line:90% Are you going from 2 to 4? 00:20:32.930 --> 00:20:34.160 align:middle line:90% What are you doing? 00:20:34.160 --> 00:20:36.560 align:middle line:90% It's such an imprecise amount. 00:20:36.560 --> 00:20:42.761 align:middle line:84% When you're going by F1 F-stop, it means going from 2 to 2.8. 00:20:42.761 --> 00:20:45.490 align:middle line:84% It's completely unnecessary [INAUDIBLE].. 00:20:45.490 --> 00:20:49.290 align:middle line:84% So just get rid of this terminology altogether. 00:20:49.290 --> 00:20:52.310 align:middle line:84% And if you really want to think about-- 00:20:52.310 --> 00:20:55.400 align:middle line:84% the reason why it's worthwhile in thinking 00:20:55.400 --> 00:21:02.360 align:middle line:84% about the ratio of diameter to the focal length 00:21:02.360 --> 00:21:04.570 align:middle line:90% is about the amount of light. 00:21:04.570 --> 00:21:07.410 align:middle line:84% And can I tell you why that's the case? 00:21:07.410 --> 00:21:11.195 align:middle line:84% So let's say I have a 25-millimeter lens with a 2.5-- 00:21:11.195 --> 00:21:14.645 align:middle line:84% sorry, 50-millimeter focal length. 00:21:14.645 --> 00:21:21.350 align:middle line:84% Or let's say I have a 5-millimeter lens 00:21:21.350 --> 00:21:27.260 align:middle line:84% with a 10-millimeter focal length. 00:21:27.260 --> 00:21:31.070 align:middle line:84% This system and this system have the same exact f-number, 00:21:31.070 --> 00:21:32.180 align:middle line:90% which is 2. 00:21:32.180 --> 00:21:34.770 align:middle line:90% Here it's 50 by 25. 00:21:34.770 --> 00:21:37.400 align:middle line:90% Here it's 10 by 5. 00:21:37.400 --> 00:21:40.440 align:middle line:84% So both of them have the same f-number. 00:21:40.440 --> 00:21:41.888 align:middle line:90% What's constant between the two? 00:21:41.888 --> 00:21:48.370 align:middle line:90% 00:21:48.370 --> 00:21:49.360 align:middle line:90% This angle here. 00:21:49.360 --> 00:21:53.527 align:middle line:90% 00:21:53.527 --> 00:21:56.450 align:middle line:84% This angle is the same in both systems. 00:21:56.450 --> 00:21:59.420 align:middle line:84% And this is critical, because when you are looking 00:21:59.420 --> 00:22:03.510 align:middle line:84% at the [? world, ?] you want to say, at the given pixel, 00:22:03.510 --> 00:22:08.340 align:middle line:84% over what [INAUDIBLE] am I [INAUDIBLE]?? 00:22:08.340 --> 00:22:10.058 align:middle line:84% And so that's a very important factor 00:22:10.058 --> 00:22:11.600 align:middle line:84% when you think about how many photons 00:22:11.600 --> 00:22:13.505 align:middle line:84% you want to capture, because the angle-- 00:22:13.505 --> 00:22:16.850 align:middle line:84% the [INAUDIBLE] angle [INAUDIBLE] from the light 00:22:16.850 --> 00:22:20.930 align:middle line:84% determines how [INAUDIBLE],, and certainly helps you to think 00:22:20.930 --> 00:22:25.910 align:middle line:90% in terms of ratios of angles. 00:22:25.910 --> 00:22:30.440 align:middle line:84% Other than that, [INAUDIBLE] terminologies [INAUDIBLE].. 00:22:30.440 --> 00:22:31.880 align:middle line:90% What happens after 4? 00:22:31.880 --> 00:22:35.090 align:middle line:90% What's the next f-number? 00:22:35.090 --> 00:22:36.140 align:middle line:90% AUDIENCE: 5.6. 00:22:36.140 --> 00:22:39.800 align:middle line:90% RAMESH RASKAR: 5.6. 00:22:39.800 --> 00:22:43.640 align:middle line:84% What do you think 4 times square root of 2 is? 00:22:43.640 --> 00:22:46.700 align:middle line:90% 00:22:46.700 --> 00:22:49.290 align:middle line:90% Slower [INAUDIBLE]. 00:22:49.290 --> 00:22:50.710 align:middle line:90% AUDIENCE: 5.64. 00:22:50.710 --> 00:22:53.920 align:middle line:84% RAMESH RASKAR: It's probably 1.4 times 4. 00:22:53.920 --> 00:22:57.510 align:middle line:90% So it's 4 point-- 00:22:57.510 --> 00:22:58.230 align:middle line:90% yeah. 00:22:58.230 --> 00:23:01.890 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:23:01.890 --> 00:23:06.267 align:middle line:84% RAMESH RASKAR: 5.64, [INAUDIBLE].. 00:23:06.267 --> 00:23:07.600 align:middle line:90% Getting more and more imprecise. 00:23:07.600 --> 00:23:10.550 align:middle line:84% And as you can imagine, as you go further away, you-- 00:23:10.550 --> 00:23:15.290 align:middle line:84% so this is the same confusion between whether a megabyte is 00:23:15.290 --> 00:23:18.810 align:middle line:90% 100 kilobytes or 124. 00:23:18.810 --> 00:23:20.850 align:middle line:84% Actually, it's even more confusing than this. 00:23:20.850 --> 00:23:25.650 align:middle line:84% And I heard a very interesting story at [INAUDIBLE].. 00:23:25.650 --> 00:23:29.580 align:middle line:84% When the economy was not going so well, 00:23:29.580 --> 00:23:33.030 align:middle line:84% in the contract they said, we will charge you 00:23:33.030 --> 00:23:36.600 align:middle line:84% this much for each megabyte or each gigabyte. 00:23:36.600 --> 00:23:39.430 align:middle line:84% And [INAUDIBLE] a kilobyte-- the difference is very small. 00:23:39.430 --> 00:23:40.950 align:middle line:90% It's less than 2%. 00:23:40.950 --> 00:23:46.020 align:middle line:84% [INAUDIBLE] larger gigabyte to terabyte. 00:23:46.020 --> 00:23:50.400 align:middle line:84% Difference between the power of 10 and power of 2 00:23:50.400 --> 00:23:52.510 align:middle line:90% is actually significant. 00:23:52.510 --> 00:23:54.990 align:middle line:84% So this woman wakes up one day and says, 00:23:54.990 --> 00:23:58.990 align:middle line:84% you know what, I'm gonna charge people by the bytes-- 00:23:58.990 --> 00:24:01.500 align:middle line:84% the [INAUDIBLE] power, not [INAUDIBLE] power. 00:24:01.500 --> 00:24:07.570 align:middle line:84% And so they [INAUDIBLE] by [INAUDIBLE].. 00:24:07.570 --> 00:24:12.330 align:middle line:84% And so the same kind of problem is creeping in here. 00:24:12.330 --> 00:24:15.150 align:middle line:84% [INAUDIBLE] square root of 2 is 1.4. 00:24:15.150 --> 00:24:18.310 align:middle line:84% It starts getting more and more confusing as you go. 00:24:18.310 --> 00:24:22.650 align:middle line:84% And then, as you know very well, after 5.6 you start jumping. 00:24:22.650 --> 00:24:28.310 align:middle line:84% You have 8, and 11, and 22, and [INAUDIBLE].. 00:24:28.310 --> 00:24:29.820 align:middle line:84% And then a [INAUDIBLE] plenty more. 00:24:29.820 --> 00:24:30.945 align:middle line:90% It just gets too confusing. 00:24:30.945 --> 00:24:34.200 align:middle line:90% So just ignore [INAUDIBLE]. 00:24:34.200 --> 00:24:36.510 align:middle line:90% It's the most confusing thing. 00:24:36.510 --> 00:24:39.110 align:middle line:84% Way down by F-stop, I think I should go ahead and approve 00:24:39.110 --> 00:24:40.066 align:middle line:90% [INAUDIBLE]. 00:24:40.066 --> 00:24:45.510 align:middle line:90% 00:24:45.510 --> 00:24:47.100 align:middle line:90% Photography school [INAUDIBLE]. 00:24:47.100 --> 00:24:50.190 align:middle line:90% 00:24:50.190 --> 00:24:53.370 align:middle line:84% So what we're trying to do here is 00:24:53.370 --> 00:24:58.850 align:middle line:84% our angle here is decided by this ratio, [INAUDIBLE] 00:24:58.850 --> 00:25:00.430 align:middle line:90% to focal length. 00:25:00.430 --> 00:25:03.000 align:middle line:84% And the angle here is this-- again, 00:25:03.000 --> 00:25:06.210 align:middle line:84% a ratio of the blob to this distance. 00:25:06.210 --> 00:25:08.130 align:middle line:90% So let's call this C1. 00:25:08.130 --> 00:25:10.590 align:middle line:90% And let's call this [INAUDIBLE]. 00:25:10.590 --> 00:25:15.120 align:middle line:90% 00:25:15.120 --> 00:25:18.750 align:middle line:84% So what you want is the ratio of e-- 00:25:18.750 --> 00:25:29.580 align:middle line:84% sorry c to d to be equal to the ratio of l to your 00:25:29.580 --> 00:25:38.180 align:middle line:84% [? losses. ?] And if that is matched, 00:25:38.180 --> 00:25:41.256 align:middle line:84% then all your blobs are going to just barely touch each other. 00:25:41.256 --> 00:25:44.150 align:middle line:84% If it's not matched, then either you'll get [INAUDIBLE] 00:25:44.150 --> 00:25:47.810 align:middle line:90% or you'll get [INAUDIBLE]. 00:25:47.810 --> 00:25:53.030 align:middle line:84% That's the basic math behind a light frame. 00:25:53.030 --> 00:25:55.880 align:middle line:84% So it's very conceptually, but as we know, 00:25:55.880 --> 00:25:58.190 align:middle line:90% it's going to block the light. 00:25:58.190 --> 00:26:02.690 align:middle line:84% And [INAUDIBLE] even talk about [INAUDIBLE].. 00:26:02.690 --> 00:26:06.578 align:middle line:84% Has anybody done, here, pinhole photography? 00:26:06.578 --> 00:26:08.120 align:middle line:84% So what are the problems you've faced 00:26:08.120 --> 00:26:10.135 align:middle line:90% in your pinhole photography? 00:26:10.135 --> 00:26:12.635 align:middle line:84% AUDIENCE: It's just, there's so little light [INAUDIBLE]---- 00:26:12.635 --> 00:26:13.310 align:middle line:84% RAMESH RASKAR: So little light, so-- 00:26:13.310 --> 00:26:14.430 align:middle line:90% AUDIENCE: --exposure time. 00:26:14.430 --> 00:26:16.700 align:middle line:84% RAMESH RASKAR: Exposure time is very long. 00:26:16.700 --> 00:26:17.930 align:middle line:90% And? 00:26:17.930 --> 00:26:19.130 align:middle line:90% And the image quality? 00:26:19.130 --> 00:26:21.050 align:middle line:84% AUDIENCE: It's blurry, because you could never 00:26:21.050 --> 00:26:22.552 align:middle line:90% [INAUDIBLE] the small holes. 00:26:22.552 --> 00:26:23.510 align:middle line:90% RAMESH RASKAR: Exactly. 00:26:23.510 --> 00:26:26.270 align:middle line:84% The image is blurred because of refraction. 00:26:26.270 --> 00:26:32.903 align:middle line:84% So if you're viewing in a pinhole, then light comes in. 00:26:32.903 --> 00:26:34.820 align:middle line:84% And it actually doesn't go in a straight line. 00:26:34.820 --> 00:26:36.266 align:middle line:90% It actually bends a little bit. 00:26:36.266 --> 00:26:40.730 align:middle line:90% 00:26:40.730 --> 00:26:44.360 align:middle line:84% And because of that, the single point in the [? world ?] 00:26:44.360 --> 00:26:46.220 align:middle line:84% doesn't map to a single point in the sensor. 00:26:46.220 --> 00:26:50.210 align:middle line:84% But it maps to a small [? blur. ?] 00:26:50.210 --> 00:26:51.920 align:middle line:84% And this is very similar to the analogy 00:26:51.920 --> 00:26:54.750 align:middle line:90% of just using a water hose. 00:26:54.750 --> 00:26:58.190 align:middle line:84% If you have a water hose, and it's open, water comes in. 00:26:58.190 --> 00:27:04.770 align:middle line:84% And you get always the same thickness, same width 00:27:04.770 --> 00:27:07.030 align:middle line:90% of the water flow, right? 00:27:07.030 --> 00:27:11.220 align:middle line:84% But as you start shifting this, eventually the water just 00:27:11.220 --> 00:27:13.722 align:middle line:90% starts spraying [INAUDIBLE]. 00:27:13.722 --> 00:27:19.410 align:middle line:84% When the size of your opening in the water hose 00:27:19.410 --> 00:27:24.077 align:middle line:84% becomes comparable to the molecules of water, 00:27:24.077 --> 00:27:25.410 align:middle line:90% it actually starts spraying out. 00:27:25.410 --> 00:27:28.100 align:middle line:90% And that's diffraction, as well. 00:27:28.100 --> 00:27:29.910 align:middle line:84% And we have the same principal here. 00:27:29.910 --> 00:27:32.365 align:middle line:84% In very simple words, we have photons coming in here 00:27:32.365 --> 00:27:33.682 align:middle line:90% with certain wavelengths. 00:27:33.682 --> 00:27:38.250 align:middle line:84% What's the wavelength of green light? 00:27:38.250 --> 00:27:39.340 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:27:39.340 --> 00:27:40.590 align:middle line:90% RAMESH RASKAR: 500 nanometers. 00:27:40.590 --> 00:27:41.548 align:middle line:90% Remember these numbers. 00:27:41.548 --> 00:27:43.990 align:middle line:90% They're very, very [INAUDIBLE]. 00:27:43.990 --> 00:27:49.458 align:middle line:90% So 400 to 700 is blue, green-- 00:27:49.458 --> 00:27:52.446 align:middle line:90% 00:27:52.446 --> 00:27:55.106 align:middle line:84% and this is nanometers [INAUDIBLE].. 00:27:55.106 --> 00:27:59.940 align:middle line:90% 00:27:59.940 --> 00:28:06.270 align:middle line:84% So if the light is 500 nanometers, which 00:28:06.270 --> 00:28:15.090 align:middle line:84% is 0.5 micrometers, and your pinhole starts [INAUDIBLE] 00:28:15.090 --> 00:28:21.690 align:middle line:84% about 1 millimeter, which is 1,000 micrometers, 00:28:21.690 --> 00:28:22.930 align:middle line:90% then you're probably OK. 00:28:22.930 --> 00:28:25.590 align:middle line:84% But as you start weighing millimeter 00:28:25.590 --> 00:28:28.200 align:middle line:84% and below-- which is what your cell phone cameras are, 00:28:28.200 --> 00:28:31.290 align:middle line:84% fortunately-- you start getting to 500 microns, which 00:28:31.290 --> 00:28:32.580 align:middle line:90% is half a millimeter. 00:28:32.580 --> 00:28:35.880 align:middle line:84% Then your wavelength is comparable to the size 00:28:35.880 --> 00:28:37.230 align:middle line:90% of your photon. 00:28:37.230 --> 00:28:39.402 align:middle line:84% And you start getting this [INAUDIBLE].. 00:28:39.402 --> 00:28:42.140 align:middle line:90% 00:28:42.140 --> 00:28:46.670 align:middle line:84% So the relationship between the size of opening 00:28:46.670 --> 00:28:48.620 align:middle line:84% and the size of the wavelength kind of 00:28:48.620 --> 00:28:50.060 align:middle line:90% decides the diffraction. 00:28:50.060 --> 00:28:51.860 align:middle line:84% And the focal length is very easy. 00:28:51.860 --> 00:28:55.690 align:middle line:90% This one here is simply-- 00:28:55.690 --> 00:29:00.274 align:middle line:90% 00:29:00.274 --> 00:29:04.010 align:middle line:84% [INAUDIBLE] I said the aperture is 8. 00:29:04.010 --> 00:29:10.091 align:middle line:84% This angle here is going to be [INAUDIBLE] in gradients 00:29:10.091 --> 00:29:11.053 align:middle line:90% [INAUDIBLE]. 00:29:11.053 --> 00:29:15.580 align:middle line:90% 00:29:15.580 --> 00:29:18.063 align:middle line:84% We won't be talking about it too much in this class-- 00:29:18.063 --> 00:29:18.730 align:middle line:90% in this lecture. 00:29:18.730 --> 00:29:24.040 align:middle line:84% [INAUDIBLE] an indication of how quickly 00:29:24.040 --> 00:29:26.510 align:middle line:90% the light is turning up. 00:29:26.510 --> 00:29:28.070 align:middle line:84% So in the worst case scenario, when 00:29:28.070 --> 00:29:33.650 align:middle line:84% you have the big hole the same size as the wavelength 00:29:33.650 --> 00:29:35.912 align:middle line:84% of light-- so let's say you created a pinhole whose 00:29:35.912 --> 00:29:37.370 align:middle line:84% width equals [INAUDIBLE] nanometer, 00:29:37.370 --> 00:29:43.080 align:middle line:84% or 0.5 micrometer, what [INAUDIBLE]?? 00:29:43.080 --> 00:29:45.280 align:middle line:84% If you're wondering, 1 gradient is 00:29:45.280 --> 00:29:50.330 align:middle line:90% how many of this, [INAUDIBLE]. 00:29:50.330 --> 00:29:53.128 align:middle line:90% So this cone will be 57 degrees. 00:29:53.128 --> 00:29:56.984 align:middle line:90% 00:29:56.984 --> 00:29:58.970 align:middle line:90% So it's part of the question. 00:29:58.970 --> 00:30:00.980 align:middle line:84% Even if you can just do calculations, 00:30:00.980 --> 00:30:04.490 align:middle line:84% and even if you plot something that's 1,001 millimeter-- 00:30:04.490 --> 00:30:07.265 align:middle line:84% so that's [INAUDIBLE] micrometers-- 00:30:07.265 --> 00:30:10.470 align:middle line:84% the angle's [INAUDIBLE] pretty wide. 00:30:10.470 --> 00:30:11.750 align:middle line:90% That's what you need to know. 00:30:11.750 --> 00:30:17.030 align:middle line:84% And so when camera makers are selling you lenses-- 00:30:17.030 --> 00:30:21.110 align:middle line:84% really crappy lenses on a mobile phone camera, 00:30:21.110 --> 00:30:23.720 align:middle line:84% but just giving you a [INAUDIBLE] resolution 00:30:23.720 --> 00:30:27.670 align:middle line:84% of 5 megapixels and 10 megapixels, 00:30:27.670 --> 00:30:32.170 align:middle line:84% it doesn't make sense because your image will be blurred. 00:30:32.170 --> 00:30:33.550 align:middle line:90% You don't need that many. 00:30:33.550 --> 00:30:42.250 align:middle line:84% So other way to think about that is, can your camera 00:30:42.250 --> 00:30:43.020 align:middle line:90% capture all this? 00:30:43.020 --> 00:30:43.922 align:middle line:90% AUDIENCE: Mm-hmm. 00:30:43.922 --> 00:30:45.730 align:middle line:90% [INAUDIBLE] 00:30:45.730 --> 00:30:49.300 align:middle line:84% RAMESH RASKAR: You should just take photos later on. 00:30:49.300 --> 00:30:55.990 align:middle line:84% If your cell phone camera has a typical aperture size of, say, 00:30:55.990 --> 00:30:59.920 align:middle line:84% 2 millimeters, that's [INAUDIBLE] 00:30:59.920 --> 00:31:02.380 align:middle line:84% and then your sensor light comes in, 00:31:02.380 --> 00:31:04.240 align:middle line:84% you're kind of zooming in [INAUDIBLE].. 00:31:04.240 --> 00:31:06.445 align:middle line:90% And it kind of spreads out. 00:31:06.445 --> 00:31:10.420 align:middle line:84% And let's say it spreads out over only about 20 micrometers. 00:31:10.420 --> 00:31:13.220 align:middle line:90% 00:31:13.220 --> 00:31:24.230 align:middle line:84% The pixel of a camera is about 5 micrometers each pixel. 00:31:24.230 --> 00:31:28.480 align:middle line:84% So if the blur is already 20 micrometer, 00:31:28.480 --> 00:31:31.180 align:middle line:84% it doesn't make sense to have a pixel that's that small. 00:31:31.180 --> 00:31:34.510 align:middle line:84% You should just have pixel that match. 00:31:34.510 --> 00:31:35.530 align:middle line:90% [INAUDIBLE] 00:31:35.530 --> 00:31:39.490 align:middle line:84% If someone's selling you a 10-megapixel camera 00:31:39.490 --> 00:31:42.610 align:middle line:84% with a 5-micron pitch, they should just 00:31:42.610 --> 00:31:47.650 align:middle line:84% give you something that's much lower resolution than that, 00:31:47.650 --> 00:31:51.910 align:middle line:84% because the numbers won't make any sense. 00:31:51.910 --> 00:31:56.140 align:middle line:84% But again, this is a typical marketing gimmick camera makers 00:31:56.140 --> 00:31:59.020 align:middle line:84% will use, where they will sell really high resolution sensors, 00:31:59.020 --> 00:32:04.560 align:middle line:84% although clearly, you cannot really capture that big 00:32:04.560 --> 00:32:07.540 align:middle line:90% [INAUDIBLE] resolution. 00:32:07.540 --> 00:32:08.040 align:middle line:90% Skip. 00:32:08.040 --> 00:32:11.260 align:middle line:90% 00:32:11.260 --> 00:32:12.770 align:middle line:90% And there's a new trend. 00:32:12.770 --> 00:32:15.820 align:middle line:84% I think camera makers have started selling [INAUDIBLE] 00:32:15.820 --> 00:32:17.230 align:middle line:90% megapixel [INAUDIBLE]. 00:32:17.230 --> 00:32:19.120 align:middle line:84% And now they have to start selling cameras 00:32:19.120 --> 00:32:20.270 align:middle line:90% with lower resolution. 00:32:20.270 --> 00:32:23.127 align:middle line:84% I think the latest Canon has lower resolution 00:32:23.127 --> 00:32:24.210 align:middle line:90% than its previous version. 00:32:24.210 --> 00:32:25.890 align:middle line:90% I believe the G6-- 00:32:25.890 --> 00:32:27.810 align:middle line:90% I think it's 12 megapixel. 00:32:27.810 --> 00:32:30.243 align:middle line:90% But at least now it's dynamic. 00:32:30.243 --> 00:32:32.050 align:middle line:84% Because they realized it doesn't make 00:32:32.050 --> 00:32:35.980 align:middle line:84% sense to just keep boosting the megapixels when 00:32:35.980 --> 00:32:38.920 align:middle line:84% the aperture of the lens is [INAUDIBLE].. 00:32:38.920 --> 00:32:41.050 align:middle line:84% So recently, I was working on a project 00:32:41.050 --> 00:32:47.750 align:middle line:84% where they wanted to capture 50 gigapixels. 00:32:47.750 --> 00:32:49.200 align:middle line:90% 50 gigapixels. 00:32:49.200 --> 00:32:51.985 align:middle line:90% 50 times 10 to the 9. 00:32:51.985 --> 00:32:55.985 align:middle line:84% This is usually 2 times [? 10 to the ?] 6 pixels. 00:32:55.985 --> 00:32:59.970 align:middle line:90% 00:32:59.970 --> 00:33:07.860 align:middle line:84% And as the megapixels increased, the size of the-- 00:33:07.860 --> 00:33:11.370 align:middle line:84% as you can see, this ratio here-- 00:33:11.370 --> 00:33:13.770 align:middle line:84% as the megapixels increased, the size of the lens 00:33:13.770 --> 00:33:17.130 align:middle line:84% required increases correspondingly. 00:33:17.130 --> 00:33:20.040 align:middle line:84% And that's why expensive cameras have much bigger lenses. 00:33:20.040 --> 00:33:21.630 align:middle line:90% AUDIENCE: [INAUDIBLE]. 00:33:21.630 --> 00:33:22.630 align:middle line:90% I had a question. 00:33:22.630 --> 00:33:25.788 align:middle line:84% [INAUDIBLE],, wouldn't it be much more 00:33:25.788 --> 00:33:28.530 align:middle line:84% effective to come up with a measure [INAUDIBLE] 00:33:28.530 --> 00:33:31.394 align:middle line:90% size of the camera [INAUDIBLE]? 00:33:31.394 --> 00:33:34.772 align:middle line:84% [INAUDIBLE] can't really [INAUDIBLE].. 00:33:34.772 --> 00:33:35.730 align:middle line:90% RAMESH RASKAR: Exactly. 00:33:35.730 --> 00:33:36.647 align:middle line:90% AUDIENCE: [INAUDIBLE]. 00:33:36.647 --> 00:33:38.730 align:middle line:84% RAMESH RASKAR: And the camera makers [INAUDIBLE].. 00:33:38.730 --> 00:33:39.330 align:middle line:90% AUDIENCE: Yes. 00:33:39.330 --> 00:33:40.860 align:middle line:84% I mean, of course it's a marketing thing. 00:33:40.860 --> 00:33:41.010 align:middle line:90% [INAUDIBLE] 00:33:41.010 --> 00:33:41.160 align:middle line:90% RAMESH RASKAR: Yeah, yeah. 00:33:41.160 --> 00:33:42.120 align:middle line:90% AUDIENCE: [INAUDIBLE]? 00:33:42.120 --> 00:33:44.872 align:middle line:90% 00:33:44.872 --> 00:33:46.080 align:middle line:90% RAMESH RASKAR: Yes, there is. 00:33:46.080 --> 00:33:49.650 align:middle line:84% So there are all these measures called modulation transfer 00:33:49.650 --> 00:33:54.100 align:middle line:84% function, and space-bandwidth product, and so on. 00:33:54.100 --> 00:33:56.650 align:middle line:84% And if you want, we can discuss about that later on. 00:33:56.650 --> 00:33:59.443 align:middle line:84% It's not so simple [INAUDIBLE] what we're doing here. 00:33:59.443 --> 00:34:01.485 align:middle line:84% But [INAUDIBLE],, yes, let's get more resolution, 00:34:01.485 --> 00:34:04.130 align:middle line:90% or, this [INAUDIBLE] resolution. 00:34:04.130 --> 00:34:05.610 align:middle line:90% Those are the laws of physics. 00:34:05.610 --> 00:34:08.290 align:middle line:84% You can have a lot of holes, but [INAUDIBLE].. 00:34:08.290 --> 00:34:14.333 align:middle line:90% 00:34:14.333 --> 00:34:16.500 align:middle line:84% So the other way to think about this is, how can you 00:34:16.500 --> 00:34:18.840 align:middle line:84% create a [INAUDIBLE] that creates a spot that's 00:34:18.840 --> 00:34:25.080 align:middle line:84% [INAUDIBLE] start decreasing your [INAUDIBLE] size 00:34:25.080 --> 00:34:27.029 align:middle line:90% for the things [INAUDIBLE]. 00:34:27.029 --> 00:34:30.949 align:middle line:90% 00:34:30.949 --> 00:34:36.380 align:middle line:84% And it's almost-- it's funny how nature works. 00:34:36.380 --> 00:34:38.120 align:middle line:84% There are always, always limits as we 00:34:38.120 --> 00:34:43.739 align:middle line:84% get closer to certain [? questions. ?] 00:34:43.739 --> 00:34:47.950 align:middle line:84% If you think about the density of water, 00:34:47.950 --> 00:34:50.694 align:middle line:84% it decreases as [INAUDIBLE] temperature. 00:34:50.694 --> 00:34:55.260 align:middle line:84% Or, at some point, [INAUDIBLE] potentially starts [INAUDIBLE] 00:34:55.260 --> 00:34:58.470 align:middle line:90% diffraction, or [INAUDIBLE]. 00:34:58.470 --> 00:35:02.760 align:middle line:84% [INAUDIBLE] where it seems like [INAUDIBLE] 00:35:02.760 --> 00:35:05.130 align:middle line:90% puts some stuff [INAUDIBLE]. 00:35:05.130 --> 00:35:11.120 align:middle line:90% 00:35:11.120 --> 00:35:13.690 align:middle line:84% So we're talking about resolution. 00:35:13.690 --> 00:35:20.360 align:middle line:84% We're talking about the ratio of the diameter to the length. 00:35:20.360 --> 00:35:23.193 align:middle line:84% There's a very minor and very subtle point here. 00:35:23.193 --> 00:35:24.610 align:middle line:84% I'll just make it for those of you 00:35:24.610 --> 00:35:27.400 align:middle line:84% who are thinking about [INAUDIBLE] 00:35:27.400 --> 00:35:29.800 align:middle line:84% focal length, effective f-number or [INAUDIBLE] number. 00:35:29.800 --> 00:35:32.530 align:middle line:84% Which is-- although we're defining it 00:35:32.530 --> 00:35:35.200 align:middle line:84% by the diameter to the focal length, 00:35:35.200 --> 00:35:38.835 align:middle line:84% the sensor is never [INAUDIBLE],, because that's really 00:35:38.835 --> 00:35:40.710 align:middle line:84% the case when you're looking at [INAUDIBLE].. 00:35:40.710 --> 00:35:42.520 align:middle line:84% When you're looking at something closer, 00:35:42.520 --> 00:35:44.560 align:middle line:84% the sensor is-- say the focal length [INAUDIBLE] 00:35:44.560 --> 00:35:50.925 align:middle line:84% millimeter, the sensor is 50-plus, like 51 or 52, maybe. 00:35:50.925 --> 00:35:54.400 align:middle line:84% But again, that's small enough that [INAUDIBLE] a lot 00:35:54.400 --> 00:35:57.640 align:middle line:84% of times, where [INAUDIBLE] like that. 00:35:57.640 --> 00:35:59.631 align:middle line:90% It's by taking [INAUDIBLE]. 00:35:59.631 --> 00:36:03.720 align:middle line:90% 00:36:03.720 --> 00:36:05.460 align:middle line:90% So we're talking about pinhole. 00:36:05.460 --> 00:36:09.990 align:middle line:84% Then we started talking about how pinholes aren't great 00:36:09.990 --> 00:36:11.820 align:middle line:90% because they lose light. 00:36:11.820 --> 00:36:15.180 align:middle line:84% Exposure ends up too long and [INAUDIBLE].. 00:36:15.180 --> 00:36:17.680 align:middle line:84% And all those problems are going to appear here as well. 00:36:17.680 --> 00:36:19.680 align:middle line:84% It's just that what we have created here 00:36:19.680 --> 00:36:27.030 align:middle line:84% is an array of 100 virtual cameras. 00:36:27.030 --> 00:36:30.180 align:middle line:84% So 100 virtual cameras where each camera has a pixel-- 00:36:30.180 --> 00:36:33.718 align:middle line:84% a resolution of [INAUDIBLE] pixels. 00:36:33.718 --> 00:36:38.780 align:middle line:84% [INAUDIBLE] one big camera into 100 cameras, each with [? 9 ?] 00:36:38.780 --> 00:36:40.100 align:middle line:90% pixels. 00:36:40.100 --> 00:36:44.060 align:middle line:84% And in the real world, it would be 10 [INAUDIBLE] cameras, 00:36:44.060 --> 00:36:47.380 align:middle line:84% [? 100/100, ?] each with a resolution of [? 9 ?] 00:36:47.380 --> 00:36:49.310 align:middle line:90% megapixels. 00:36:49.310 --> 00:36:51.530 align:middle line:84% That's what makes it interesting. 00:36:51.530 --> 00:36:55.422 align:middle line:84% From one camera [INAUDIBLE] 100,000 angles. 00:36:55.422 --> 00:36:57.380 align:middle line:84% And as we saw last time, you can do it at home. 00:36:57.380 --> 00:36:58.342 align:middle line:90% [INAUDIBLE] 00:36:58.342 --> 00:37:02.190 align:middle line:90% 00:37:02.190 --> 00:37:06.710 align:middle line:84% So just like in regular cameras, we don't use pinholes. 00:37:06.710 --> 00:37:07.918 align:middle line:90% We use lenses. 00:37:07.918 --> 00:37:09.710 align:middle line:84% You want to replace each of these pinholes, 00:37:09.710 --> 00:37:11.562 align:middle line:90% now, with lenses. 00:37:11.562 --> 00:37:14.090 align:middle line:84% And that's why we call it [INAUDIBLE].. 00:37:14.090 --> 00:37:17.300 align:middle line:84% So I'm just going to draw on top of this with a different color. 00:37:17.300 --> 00:37:19.730 align:middle line:84% Tell me if it gets too complicated. 00:37:19.730 --> 00:37:21.531 align:middle line:84% I'm just going to draw the landscape. 00:37:21.531 --> 00:37:25.770 align:middle line:90% 00:37:25.770 --> 00:37:29.530 align:middle line:84% And it's going to do almost the same task. 00:37:29.530 --> 00:37:35.160 align:middle line:84% We know that at the center of the lens, [INAUDIBLE].. 00:37:35.160 --> 00:37:41.410 align:middle line:84% Because remember, a lens is made up of [INAUDIBLE].. 00:37:41.410 --> 00:37:44.440 align:middle line:84% And the middle one is just a sheet of glass. 00:37:44.440 --> 00:37:47.900 align:middle line:84% So when it goes to the center, nothing changes. 00:37:47.900 --> 00:37:50.420 align:middle line:90% [INAUDIBLE] 00:37:50.420 --> 00:37:53.560 align:middle line:84% When you use a lens [INAUDIBLE],, we can do the same thing. 00:37:53.560 --> 00:37:56.350 align:middle line:90% But there's one very special-- 00:37:56.350 --> 00:37:58.750 align:middle line:84% two, actually, very special constraints 00:37:58.750 --> 00:38:04.050 align:middle line:84% we have to achieve to make this happen. 00:38:04.050 --> 00:38:07.850 align:middle line:84% Because it's a lens and we're trying to form an image. 00:38:07.850 --> 00:38:10.350 align:middle line:84% When we're converting light to the camera, what we're really 00:38:10.350 --> 00:38:16.992 align:middle line:84% doing is forming an image of the lens on the sensor. 00:38:16.992 --> 00:38:18.450 align:middle line:84% You remember, we heard this concept 00:38:18.450 --> 00:38:20.880 align:middle line:84% of imaginary plane, a point in 3D 00:38:20.880 --> 00:38:24.600 align:middle line:84% as well as pointing a location on the-- 00:38:24.600 --> 00:38:28.052 align:middle line:84% sensor position is what [INAUDIBLE] the picture, 00:38:28.052 --> 00:38:29.575 align:middle line:90% the corresponding [INAUDIBLE]. 00:38:29.575 --> 00:38:31.200 align:middle line:84% And here, what you want to do is create 00:38:31.200 --> 00:38:35.340 align:middle line:84% a lens which forms an image of the lens all 00:38:35.340 --> 00:38:39.950 align:middle line:84% [? the same size. ?] [INAUDIBLE] this focal length 00:38:39.950 --> 00:38:43.980 align:middle line:84% and its distances in such a way that if I put a point here, 00:38:43.980 --> 00:38:47.750 align:middle line:84% I get a sharp image of the point here. 00:38:47.750 --> 00:38:49.850 align:middle line:84% Because in other words, [INAUDIBLE].. 00:38:49.850 --> 00:38:53.680 align:middle line:84% And each detail here has to map to 1 pixel exactly. 00:38:53.680 --> 00:38:56.907 align:middle line:90% 00:38:56.907 --> 00:39:00.110 align:middle line:84% So that makes it extremely difficult. 00:39:00.110 --> 00:39:03.770 align:middle line:84% Because now we're almost in a microscopic point. 00:39:03.770 --> 00:39:07.640 align:middle line:84% And [? I do ?] create these lenses extremely high quality 00:39:07.640 --> 00:39:11.870 align:middle line:84% so that an image of a point can-- at about 50 millimeters 00:39:11.870 --> 00:39:16.550 align:middle line:84% is formed at some very small distance, usually 1 [INAUDIBLE] 00:39:16.550 --> 00:39:18.170 align:middle line:90% or 1 millimeter. 00:39:18.170 --> 00:39:22.410 align:middle line:84% It's about 500 microns [INAUDIBLE].. 00:39:22.410 --> 00:39:23.810 align:middle line:84% So this becomes very challenging. 00:39:23.810 --> 00:39:25.790 align:middle line:84% This is a very specific constraint. 00:39:25.790 --> 00:39:29.540 align:middle line:84% There's exactly one plane in which I can put the sensor. 00:39:29.540 --> 00:39:32.442 align:middle line:84% If I put it too far here, then the image 00:39:32.442 --> 00:39:34.150 align:middle line:84% of the lens on the sensor will be blurry. 00:39:34.150 --> 00:39:37.346 align:middle line:84% If I put it too close, again, the image will be blurry. 00:39:37.346 --> 00:39:41.080 align:middle line:90% 00:39:41.080 --> 00:39:45.993 align:middle line:84% And that very special constraint makes 00:39:45.993 --> 00:39:48.160 align:middle line:84% building a lightfield camera out of lens [INAUDIBLE] 00:39:48.160 --> 00:39:52.201 align:middle line:84% is extremely challenging and very expensive. 00:39:52.201 --> 00:39:55.350 align:middle line:90% [INAUDIBLE] 00:39:55.350 --> 00:39:57.470 align:middle line:90% So let's say you-- 00:39:57.470 --> 00:40:00.590 align:middle line:84% by the way, all these things we talk about where if you put 00:40:00.590 --> 00:40:02.210 align:middle line:90% the lens stack-- 00:40:02.210 --> 00:40:05.612 align:middle line:84% you can put the lens a little bit further out 00:40:05.612 --> 00:40:07.820 align:middle line:84% and change its focal length so that this one is still 00:40:07.820 --> 00:40:09.260 align:middle line:90% imaged here. 00:40:09.260 --> 00:40:12.150 align:middle line:84% But conceptually, it's to the pinhole. 00:40:12.150 --> 00:40:14.040 align:middle line:84% So if you put the lenses over here 00:40:14.040 --> 00:40:18.710 align:middle line:84% and the ratios of this to this is not matched to this to this, 00:40:18.710 --> 00:40:19.958 align:middle line:90% then you'll get overlap. 00:40:19.958 --> 00:40:22.250 align:middle line:84% And if you put it too close and change the focal length 00:40:22.250 --> 00:40:25.250 align:middle line:84% accordingly, you'll get dark spots [INAUDIBLE].. 00:40:25.250 --> 00:40:27.625 align:middle line:84% So we still have to worry about a standard set of issues. 00:40:27.625 --> 00:40:31.430 align:middle line:90% 00:40:31.430 --> 00:40:37.460 align:middle line:84% So this additional thing you have to do, which won't look-- 00:40:37.460 --> 00:40:38.460 align:middle line:90% a lot of people forget-- 00:40:38.460 --> 00:40:40.640 align:middle line:84% which is, if there's some points-- 00:40:40.640 --> 00:40:58.530 align:middle line:90% 00:40:58.530 --> 00:41:02.420 align:middle line:84% there's some point here out in the world [INAUDIBLE] again 00:41:02.420 --> 00:41:05.560 align:middle line:90% [INAUDIBLE]. 00:41:05.560 --> 00:41:07.420 align:middle line:84% And what we're trying to do is, this point 00:41:07.420 --> 00:41:11.040 align:middle line:90% is in focus over here. 00:41:11.040 --> 00:41:14.295 align:middle line:84% And then we're going to chop this into nine [INAUDIBLE],, 00:41:14.295 --> 00:41:19.290 align:middle line:84% and [INAUDIBLE] plus 4, [INAUDIBLE] minus 4. 00:41:19.290 --> 00:41:20.940 align:middle line:84% And the one that goes to the center 00:41:20.940 --> 00:41:23.750 align:middle line:84% gets divided too plus [INAUDIBLE].. 00:41:23.750 --> 00:41:25.350 align:middle line:90% That is very focused. 00:41:25.350 --> 00:41:28.380 align:middle line:84% And what we have done is we have captured the lightstream 00:41:28.380 --> 00:41:29.640 align:middle line:90% of [INAUDIBLE]. 00:41:29.640 --> 00:41:34.250 align:middle line:90% 00:41:34.250 --> 00:41:39.740 align:middle line:84% If I want to create a lightfield, I [INAUDIBLE] 00:41:39.740 --> 00:41:43.280 align:middle line:90% differently over here. 00:41:43.280 --> 00:41:44.620 align:middle line:90% The question is, can we-- 00:41:44.620 --> 00:41:47.660 align:middle line:84% what's happened with-- remember, we made this claim 00:41:47.660 --> 00:41:50.120 align:middle line:84% that if I capture this 4D lightfield, 00:41:50.120 --> 00:41:52.160 align:middle line:90% I can do anything I want. 00:41:52.160 --> 00:41:53.120 align:middle line:90% I can focus here. 00:41:53.120 --> 00:41:54.650 align:middle line:90% I can focus there. 00:41:54.650 --> 00:41:55.520 align:middle line:90% I can use the light. 00:41:55.520 --> 00:41:57.050 align:middle line:90% I can change the aperture size. 00:41:57.050 --> 00:41:58.580 align:middle line:90% I can change the focal length. 00:41:58.580 --> 00:42:03.000 align:middle line:84% I can do anything I want if I capture that [INAUDIBLE].. 00:42:03.000 --> 00:42:09.230 align:middle line:84% So one question to be thinking about when 00:42:09.230 --> 00:42:13.390 align:middle line:84% you are doing this assignment is [INAUDIBLE] capture 00:42:13.390 --> 00:42:15.350 align:middle line:84% the light for one particular plane, 00:42:15.350 --> 00:42:17.636 align:middle line:84% how do you recreate it [INAUDIBLE],, 00:42:17.636 --> 00:42:23.127 align:middle line:84% either by refocusing it or [INAUDIBLE]?? 00:42:23.127 --> 00:42:24.960 align:middle line:84% What's the difference between this situation 00:42:24.960 --> 00:42:27.590 align:middle line:84% and the assignment that you're doing? 00:42:27.590 --> 00:42:31.060 align:middle line:90% 00:42:31.060 --> 00:42:34.990 align:middle line:84% Because [INAUDIBLE] cameras, and maybe last time 00:42:34.990 --> 00:42:38.650 align:middle line:84% we realized that the lens is what? 00:42:38.650 --> 00:42:45.850 align:middle line:84% It's the area of impulse with [INAUDIBLE].. 00:42:45.850 --> 00:42:50.220 align:middle line:84% I can chop the lens as [INAUDIBLE].. 00:42:50.220 --> 00:42:55.652 align:middle line:90% 00:42:55.652 --> 00:42:59.390 align:middle line:84% You can take off the lens and you can chop it up. 00:42:59.390 --> 00:43:07.595 align:middle line:84% And you can treat it the same as these pinholes, 00:43:07.595 --> 00:43:10.750 align:middle line:84% which corresponding to something next to it. 00:43:10.750 --> 00:43:14.718 align:middle line:90% 00:43:14.718 --> 00:43:16.720 align:middle line:84% In this case, just a sheet of glass. 00:43:16.720 --> 00:43:21.610 align:middle line:90% 00:43:21.610 --> 00:43:24.190 align:middle line:90% It's a pinhole plus a prism. 00:43:24.190 --> 00:43:26.710 align:middle line:84% When you take a picture with a camera array, unfortunately, 00:43:26.710 --> 00:43:29.020 align:middle line:84% you're going to have to do that computation. 00:43:29.020 --> 00:43:30.625 align:middle line:84% All you have is a set of cameras. 00:43:30.625 --> 00:43:34.764 align:middle line:90% 00:43:34.764 --> 00:43:35.752 align:middle line:90% AUDIENCE: [INAUDIBLE]. 00:43:35.752 --> 00:43:37.480 align:middle line:90% RAMESH RASKAR: Thanks. 00:43:37.480 --> 00:43:39.010 align:middle line:84% All you have is a set of cameras. 00:43:39.010 --> 00:43:41.350 align:middle line:90% You don't have this prism. 00:43:41.350 --> 00:43:45.310 align:middle line:84% And mathematically, you're going to shift the image, which 00:43:45.310 --> 00:43:47.100 align:middle line:90% is equivalent [INAUDIBLE]. 00:43:47.100 --> 00:43:48.850 align:middle line:84% So you're going to take [INAUDIBLE] images 00:43:48.850 --> 00:43:51.100 align:middle line:84% and you're going to shift them by plus 2 pixels, 00:43:51.100 --> 00:43:55.490 align:middle line:84% plus 1 pixel, 0 pixel, minus 1 pixel, minus 2 pixel. 00:43:55.490 --> 00:43:59.320 align:middle line:84% And that's the same as putting a set of prisms. 00:43:59.320 --> 00:44:00.250 align:middle line:90% Is this analogy clear? 00:44:00.250 --> 00:44:05.320 align:middle line:90% 00:44:05.320 --> 00:44:10.360 align:middle line:84% So once we have done all this, there's 00:44:10.360 --> 00:44:14.320 align:middle line:84% two classic ways of thinking about capturing lightfields. 00:44:14.320 --> 00:44:20.078 align:middle line:84% Again, 1908 is when this idea started coming up, [INAUDIBLE].. 00:44:20.078 --> 00:44:22.645 align:middle line:84% But the practical solutions came much later. 00:44:22.645 --> 00:44:24.615 align:middle line:84% Remember, back then, it [INAUDIBLE].. 00:44:24.615 --> 00:44:29.170 align:middle line:90% 00:44:29.170 --> 00:44:33.766 align:middle line:84% Meanwhile, a third solution came on just two years ago, 00:44:33.766 --> 00:44:35.877 align:middle line:90% whilst capturing lightfields. 00:44:35.877 --> 00:44:38.210 align:middle line:84% And we're going to look at that for the next few minutes 00:44:38.210 --> 00:44:41.593 align:middle line:84% and then switch over to [INAUDIBLE].. 00:44:41.593 --> 00:44:43.760 align:middle line:84% Anybody has thoughts on some other ways of capturing 00:44:43.760 --> 00:44:47.260 align:middle line:90% the light [INAUDIBLE]? 00:44:47.260 --> 00:44:48.150 align:middle line:90% Yes? 00:44:48.150 --> 00:44:49.545 align:middle line:90% AUDIENCE: [INAUDIBLE]. 00:44:49.545 --> 00:44:57.658 align:middle line:90% 00:44:57.658 --> 00:44:58.700 align:middle line:90% RAMESH RASKAR: Excellent. 00:44:58.700 --> 00:44:59.420 align:middle line:90% Excellent. 00:44:59.420 --> 00:45:03.090 align:middle line:84% So you [INAUDIBLE] lots of [? hats ?] to go around this. 00:45:03.090 --> 00:45:06.110 align:middle line:84% So one very interesting type these guys came up 00:45:06.110 --> 00:45:09.350 align:middle line:84% with was they say, I don't want to put anything close 00:45:09.350 --> 00:45:11.150 align:middle line:90% to the sensor-- 00:45:11.150 --> 00:45:13.130 align:middle line:90% just a traditional camera. 00:45:13.130 --> 00:45:15.683 align:middle line:90% And [INAUDIBLE] this. 00:45:15.683 --> 00:45:17.100 align:middle line:84% I'm going to do this part up here. 00:45:17.100 --> 00:45:28.347 align:middle line:90% 00:45:28.347 --> 00:45:31.310 align:middle line:84% Because all I care about is this pixel, 00:45:31.310 --> 00:45:34.300 align:middle line:84% I want to know how each of these rays 00:45:34.300 --> 00:45:37.010 align:middle line:84% [INAUDIBLE] what is the radius of each [INAUDIBLE]?? 00:45:37.010 --> 00:45:41.360 align:middle line:84% So all I will do is I will block this part of the lens. 00:45:41.360 --> 00:45:44.900 align:middle line:84% [INAUDIBLE] and that will give you 00:45:44.900 --> 00:45:46.490 align:middle line:90% this direction for each pixel. 00:45:46.490 --> 00:45:49.190 align:middle line:90% 00:45:49.190 --> 00:45:53.990 align:middle line:84% Then I take the second photo, but I will block everything 00:45:53.990 --> 00:45:55.380 align:middle line:90% except this region. 00:45:55.380 --> 00:45:59.090 align:middle line:84% [? 84. ?] And if I take nine such photos, 00:45:59.090 --> 00:46:05.296 align:middle line:84% then [INAUDIBLE] getting 100 times 9, what will I get? 00:46:05.296 --> 00:46:10.140 align:middle line:84% [INAUDIBLE] each has a resolution of 900 [INAUDIBLE].. 00:46:10.140 --> 00:46:16.830 align:middle line:84% So we get 900 [INAUDIBLE] this whole time. 00:46:16.830 --> 00:46:22.870 align:middle line:84% And this is a single [INAUDIBLE] and 9 times [INAUDIBLE].. 00:46:22.870 --> 00:46:25.982 align:middle line:90% So this is what [INAUDIBLE]. 00:46:25.982 --> 00:46:29.250 align:middle line:84% And they came up with better [INAUDIBLE],, 00:46:29.250 --> 00:46:30.460 align:middle line:90% but better than this class. 00:46:30.460 --> 00:46:42.040 align:middle line:90% 00:46:42.040 --> 00:46:44.920 align:middle line:84% But you still learn it's not a new matter, 00:46:44.920 --> 00:46:47.830 align:middle line:84% because it's using the same basic concept that [INAUDIBLE].. 00:46:47.830 --> 00:46:54.991 align:middle line:90% 00:46:54.991 --> 00:46:56.800 align:middle line:90% You have something of your own? 00:46:56.800 --> 00:46:58.220 align:middle line:90% AUDIENCE: [INAUDIBLE]. 00:46:58.220 --> 00:46:59.220 align:middle line:90% RAMESH RASKAR: Yeah, OK. 00:46:59.220 --> 00:47:01.590 align:middle line:90% 00:47:01.590 --> 00:47:02.090 align:middle line:90% What else? 00:47:02.090 --> 00:47:04.370 align:middle line:90% Think about all the-- 00:47:04.370 --> 00:47:06.110 align:middle line:84% that document online, that presentation 00:47:06.110 --> 00:47:08.185 align:middle line:84% about how to come up with new ideas. 00:47:08.185 --> 00:47:09.560 align:middle line:84% One way to think about it-- think 00:47:09.560 --> 00:47:12.230 align:middle line:84% about all the cameras you know, all the [INAUDIBLE].. 00:47:12.230 --> 00:47:13.760 align:middle line:84% On a camera, you have [INAUDIBLE] 00:47:13.760 --> 00:47:17.600 align:middle line:84% exposure time, focus, focal length, 00:47:17.600 --> 00:47:20.090 align:middle line:90% moving the camera, wavelength. 00:47:20.090 --> 00:47:23.750 align:middle line:84% Think about all the ways you can use those parameters 00:47:23.750 --> 00:47:26.277 align:middle line:90% to reach the scope. 00:47:26.277 --> 00:47:28.700 align:middle line:90% [INAUDIBLE] 00:47:28.700 --> 00:47:30.770 align:middle line:90% This is called x [INAUDIBLE]. 00:47:30.770 --> 00:47:33.002 align:middle line:90% And you know what it means. 00:47:33.002 --> 00:47:33.560 align:middle line:90% Yeah? 00:47:33.560 --> 00:47:34.900 align:middle line:90% AUDIENCE: What if you got a-- 00:47:34.900 --> 00:47:36.692 align:middle line:84% so you're shooting out of the [INAUDIBLE].. 00:47:36.692 --> 00:47:42.453 align:middle line:90% 00:47:42.453 --> 00:47:44.370 align:middle line:84% RAMESH RASKAR: Why does it have to be a thing? 00:47:44.370 --> 00:47:44.790 align:middle line:90% AUDIENCE: Yeah. 00:47:44.790 --> 00:47:45.960 align:middle line:84% RAMESH RASKAR: [INAUDIBLE] something else. 00:47:45.960 --> 00:47:46.470 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:47:46.470 --> 00:47:46.890 align:middle line:90% RAMESH RASKAR: Exactly. 00:47:46.890 --> 00:47:48.490 align:middle line:90% What [INAUDIBLE] should it be? 00:47:48.490 --> 00:47:49.780 align:middle line:90% AUDIENCE: [INAUDIBLE]. 00:47:49.780 --> 00:47:54.006 align:middle line:90% 00:47:54.006 --> 00:47:56.650 align:middle line:84% RAMESH RASKAR: Like on some kind of a concave or [INAUDIBLE] 00:47:56.650 --> 00:47:57.556 align:middle line:90% sphere. 00:47:57.556 --> 00:48:00.460 align:middle line:90% AUDIENCE: A similar [INAUDIBLE]. 00:48:00.460 --> 00:48:03.490 align:middle line:84% RAMESH RASKAR: Remember, CAT scan machine works 00:48:03.490 --> 00:48:05.620 align:middle line:84% on the most simplest [INAUDIBLE].. 00:48:05.620 --> 00:48:07.800 align:middle line:84% And a CAT scan machine, it is [INAUDIBLE].. 00:48:07.800 --> 00:48:13.830 align:middle line:84% [INAUDIBLE],, Or even thinking of doing 00:48:13.830 --> 00:48:19.410 align:middle line:84% some non-interfering optical communication. 00:48:19.410 --> 00:48:22.430 align:middle line:90% [INAUDIBLE] interested in this. 00:48:22.430 --> 00:48:23.800 align:middle line:90% You will do the same thing. 00:48:23.800 --> 00:48:25.800 align:middle line:84% [INAUDIBLE] are coming from different directions 00:48:25.800 --> 00:48:32.040 align:middle line:84% [INAUDIBLE] them, as we told them [INAUDIBLE].. 00:48:32.040 --> 00:48:35.580 align:middle line:84% We'll put the area of angles [INAUDIBLE].. 00:48:35.580 --> 00:48:38.760 align:middle line:90% [INAUDIBLE] a bigger project. 00:48:38.760 --> 00:48:41.712 align:middle line:90% 00:48:41.712 --> 00:48:43.188 align:middle line:90% AUDIENCE: [INAUDIBLE]. 00:48:43.188 --> 00:48:46.190 align:middle line:90% 00:48:46.190 --> 00:48:48.650 align:middle line:84% RAMESH RASKAR: So let's think about this third solution, 00:48:48.650 --> 00:48:52.730 align:middle line:84% which is a relatively simple idea to explain, actually. 00:48:52.730 --> 00:48:55.280 align:middle line:84% What we're going to do is not use lens [INAUDIBLE],, 00:48:55.280 --> 00:48:57.045 align:middle line:90% but use a [INAUDIBLE]. 00:48:57.045 --> 00:48:59.428 align:middle line:90% 00:48:59.428 --> 00:49:01.220 align:middle line:84% I'll show you how this works, and then I'll 00:49:01.220 --> 00:49:02.720 align:middle line:90% explain how it works. 00:49:02.720 --> 00:49:23.560 align:middle line:90% 00:49:23.560 --> 00:49:26.920 align:middle line:84% All I'm going to do is place a mask-- instead 00:49:26.920 --> 00:49:30.160 align:middle line:84% of a pinhole array, we're going to place the mask that 00:49:30.160 --> 00:49:33.380 align:middle line:90% has certain consequences. 00:49:33.380 --> 00:49:35.282 align:middle line:90% So here, it's a pinhole array. 00:49:35.282 --> 00:49:40.120 align:middle line:84% They're going to be, basically, some [INAUDIBLE].. 00:49:40.120 --> 00:49:43.260 align:middle line:84% It's going to have some strange effect. 00:49:43.260 --> 00:49:49.590 align:middle line:84% You can bring this [INAUDIBLE] and put it [INAUDIBLE].. 00:49:49.590 --> 00:49:55.340 align:middle line:84% And if you do that, it turns out, 00:49:55.340 --> 00:49:57.470 align:middle line:84% you'll get pictures that look like this. 00:49:57.470 --> 00:50:00.710 align:middle line:90% 00:50:00.710 --> 00:50:03.060 align:middle line:84% And if you try to force it out of focus, 00:50:03.060 --> 00:50:04.810 align:middle line:84% you have this really strange [INAUDIBLE].. 00:50:04.810 --> 00:50:07.300 align:middle line:84% Part [INAUDIBLE] focus, it looks like light has 00:50:07.300 --> 00:50:09.496 align:middle line:90% been [INAUDIBLE] a little bit. 00:50:09.496 --> 00:50:11.410 align:middle line:90% But they're the same I think. 00:50:11.410 --> 00:50:12.384 align:middle line:90% [INAUDIBLE] 00:50:12.384 --> 00:50:15.800 align:middle line:90% 00:50:15.800 --> 00:50:20.480 align:middle line:84% But for out of focus, it's this really strange [INAUDIBLE].. 00:50:20.480 --> 00:50:23.810 align:middle line:84% And I'll summarize how it's computed 00:50:23.810 --> 00:50:26.420 align:middle line:84% and then come back and explain why it works. 00:50:26.420 --> 00:50:31.580 align:middle line:84% And all we're going to do is take a traditional camera. 00:50:31.580 --> 00:50:32.450 align:middle line:90% [INAUDIBLE] 00:50:32.450 --> 00:50:35.810 align:middle line:84% You could [? take ?] any image, in general, 00:50:35.810 --> 00:50:38.220 align:middle line:84% and just take its Fourier transform, 00:50:38.220 --> 00:50:41.870 align:middle line:84% which is what JPEG compression will do as a [INAUDIBLE] step. 00:50:41.870 --> 00:50:44.750 align:middle line:84% You realize that most of the energy in this Fourier 00:50:44.750 --> 00:50:49.140 align:middle line:84% transform is in the low frequencies including you put 00:50:49.140 --> 00:50:50.390 align:middle line:90% low frequencies in the center. 00:50:50.390 --> 00:50:53.572 align:middle line:84% One day you will [INAUDIBLE] center [INAUDIBLE].. 00:50:53.572 --> 00:50:58.000 align:middle line:90% 00:50:58.000 --> 00:51:02.140 align:middle line:84% So most of the energy is in low frequencies, 00:51:02.140 --> 00:51:04.160 align:middle line:90% low spatial frequencies. 00:51:04.160 --> 00:51:06.310 align:middle line:84% So in the center, you have the DC component, which 00:51:06.310 --> 00:51:09.490 align:middle line:90% is the average of all pixels. 00:51:09.490 --> 00:51:12.440 align:middle line:84% And then you have first frequency, 00:51:12.440 --> 00:51:15.100 align:middle line:84% which is how many [INAUDIBLE] you can put in a dimension, 00:51:15.100 --> 00:51:16.966 align:middle line:90% and so on. 00:51:16.966 --> 00:51:18.960 align:middle line:84% Most of the [INAUDIBLE] over here, 00:51:18.960 --> 00:51:24.200 align:middle line:84% if you place this very high frequency mask where 00:51:24.200 --> 00:51:27.100 align:middle line:84% the pinhole layer was, and then [INAUDIBLE] 00:51:27.100 --> 00:51:29.740 align:middle line:84% Fourier transformed, it looks really, really strange. 00:51:29.740 --> 00:51:34.420 align:middle line:84% It has [INAUDIBLE] energy in high frequencies, as well. 00:51:34.420 --> 00:51:36.610 align:middle line:84% And those of you who are used to looking 00:51:36.610 --> 00:51:42.290 align:middle line:84% at oscilloscope or radio frequencies and so on-- 00:51:42.290 --> 00:51:44.950 align:middle line:84% if you just capture the radiowaves, all the radio 00:51:44.950 --> 00:51:47.007 align:middle line:84% stations, and look at the spectrum, 00:51:47.007 --> 00:51:48.590 align:middle line:84% it will also look something like this. 00:51:48.590 --> 00:51:51.430 align:middle line:84% It will have some carriers in the middle. 00:51:51.430 --> 00:51:54.580 align:middle line:84% And those will get [INAUDIBLE] around the carrier. 00:51:54.580 --> 00:51:56.650 align:middle line:90% And there will be a [INAUDIBLE]. 00:51:56.650 --> 00:52:00.310 align:middle line:84% So a 99 megahertz station is transmitting its audio 00:52:00.310 --> 00:52:06.220 align:middle line:84% over [INAUDIBLE] stations and transmitting the audio 00:52:06.220 --> 00:52:07.805 align:middle line:90% [INAUDIBLE] and so forth. 00:52:07.805 --> 00:52:09.180 align:middle line:84% So this is what the spectrum will 00:52:09.180 --> 00:52:15.130 align:middle line:84% look like if you just capture any kind of [INAUDIBLE] signal. 00:52:15.130 --> 00:52:17.860 align:middle line:84% And something similar is happening here. 00:52:17.860 --> 00:52:22.802 align:middle line:84% And I'll come back to this and explain how this example works. 00:52:22.802 --> 00:52:27.230 align:middle line:84% So that-- and what we're going to do eventually 00:52:27.230 --> 00:52:29.940 align:middle line:84% is take the truly Fourier transform, 00:52:29.940 --> 00:52:33.800 align:middle line:84% which looks like this, and we're going to shift. 00:52:33.800 --> 00:52:37.160 align:middle line:84% We're going to take this 2D wall and make it 00:52:37.160 --> 00:52:42.470 align:middle line:90% into a 4D hypercube. 00:52:42.470 --> 00:52:46.050 align:middle line:84% And we're simply going to take this inverse Fourier transform 00:52:46.050 --> 00:52:49.850 align:middle line:84% and recover all those images that you 00:52:49.850 --> 00:52:53.510 align:middle line:84% will have captured if you placed the camera 00:52:53.510 --> 00:52:55.460 align:middle line:90% at different positions. 00:52:55.460 --> 00:52:58.010 align:middle line:84% It's as if we have taken the lens 00:52:58.010 --> 00:53:01.160 align:middle line:84% and split it into 81 different cameras. 00:53:01.160 --> 00:53:03.820 align:middle line:90% [INAUDIBLE] 00:53:03.820 --> 00:53:07.640 align:middle line:84% As you can imagine, from these 81 cameras 00:53:07.640 --> 00:53:11.450 align:middle line:84% we have captured-- so for every one of those slices 00:53:11.450 --> 00:53:14.210 align:middle line:84% on the lens, we have captured an image that's 00:53:14.210 --> 00:53:18.050 align:middle line:90% only 200 pixels by 200 pixels. 00:53:18.050 --> 00:53:21.210 align:middle line:84% But we have created 81 such pictures. 00:53:21.210 --> 00:53:23.870 align:middle line:84% And so here you are just seeing some 00:53:23.870 --> 00:53:29.090 align:middle line:84% of those different pictures, a few [INAUDIBLE] pictures. 00:53:29.090 --> 00:53:31.970 align:middle line:90% But using this 81-camera array-- 00:53:31.970 --> 00:53:35.750 align:middle line:84% the box I showed you last time was a 5 by 5 camera array. 00:53:35.750 --> 00:53:38.330 align:middle line:84% And you can think of this as an 81-camera array. 00:53:38.330 --> 00:53:40.130 align:middle line:84% But we didn't build a new device. 00:53:40.130 --> 00:53:43.900 align:middle line:84% It's just an ordinary SLR camera we didn't format. 00:53:43.900 --> 00:53:45.710 align:middle line:84% And we just placed the smart sensor. 00:53:45.710 --> 00:53:47.155 align:middle line:90% And certainly, we have-- 00:53:47.155 --> 00:53:50.610 align:middle line:84% for two extra dollars, we have now 81 cameras. 00:53:50.610 --> 00:53:52.940 align:middle line:84% But of course, each of them is way more efficient. 00:53:52.940 --> 00:53:57.786 align:middle line:84% AUDIENCE: What's the advantage of this mask to the pinhole-- 00:53:57.786 --> 00:53:59.423 align:middle line:90% or pinhole mask? 00:53:59.423 --> 00:54:00.840 align:middle line:84% RAMESH RASKAR: So the question is, 00:54:00.840 --> 00:54:04.865 align:middle line:84% what's the benefit of this type of mask over a pinhole array, 00:54:04.865 --> 00:54:06.075 align:middle line:90% or some [INAUDIBLE]? 00:54:06.075 --> 00:54:08.850 align:middle line:84% What are the disadvantages of pinhole arrays? 00:54:08.850 --> 00:54:11.180 align:middle line:84% AUDIENCE: More light is going to [INAUDIBLE].. 00:54:11.180 --> 00:54:12.055 align:middle line:90% RAMESH RASKAR: Right. 00:54:12.055 --> 00:54:14.270 align:middle line:84% So it's more light, because the-- 00:54:14.270 --> 00:54:17.665 align:middle line:84% almost 50% of the light will go through. 00:54:17.665 --> 00:54:20.040 align:middle line:84% So previously, we know the pinhole light is-- very little 00:54:20.040 --> 00:54:21.385 align:middle line:90% light goes through. 00:54:21.385 --> 00:54:23.130 align:middle line:84% What's the second problem [INAUDIBLE]?? 00:54:23.130 --> 00:54:23.630 align:middle line:90% Yeah? 00:54:23.630 --> 00:54:25.088 align:middle line:84% AUDIENCE: Because the image you get 00:54:25.088 --> 00:54:28.890 align:middle line:84% through a pinhole [INAUDIBLE] is not established [INAUDIBLE].. 00:54:28.890 --> 00:54:30.950 align:middle line:90% 00:54:30.950 --> 00:54:31.908 align:middle line:90% RAMESH RASKAR: Exactly. 00:54:31.908 --> 00:54:34.520 align:middle line:84% If you [INAUDIBLE] diffraction arrays. 00:54:34.520 --> 00:54:36.520 align:middle line:84% Even here, you have some diffraction [INAUDIBLE] 00:54:36.520 --> 00:54:38.780 align:middle line:90% above the [INAUDIBLE]. 00:54:38.780 --> 00:54:41.150 align:middle line:84% AUDIENCE: So it's basically assimilating. 00:54:41.150 --> 00:54:42.230 align:middle line:90% And that's not very-- 00:54:42.230 --> 00:54:42.560 align:middle line:90% RAMESH RASKAR: Exactly. 00:54:42.560 --> 00:54:42.890 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:54:42.890 --> 00:54:44.307 align:middle line:84% RAMESH RASKAR: Well, [INAUDIBLE].. 00:54:44.307 --> 00:54:48.380 align:middle line:84% What you capture looks like almost not our image, 00:54:48.380 --> 00:54:51.537 align:middle line:84% except those very strange effects are called [INAUDIBLE].. 00:54:51.537 --> 00:54:53.120 align:middle line:84% AUDIENCE: So but are you saying you're 00:54:53.120 --> 00:54:57.080 align:middle line:84% applying the inverse transform to the small sessions 00:54:57.080 --> 00:54:59.460 align:middle line:90% of this spectrum, right? 00:54:59.460 --> 00:55:00.710 align:middle line:90% RAMESH RASKAR: No, not really. 00:55:00.710 --> 00:55:01.090 align:middle line:90% Not really. 00:55:01.090 --> 00:55:01.965 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:55:01.965 --> 00:55:04.040 align:middle line:84% RAMESH RASKAR: We'll take this full 2D image. 00:55:04.040 --> 00:55:06.810 align:middle line:84% And we're going to [INAUDIBLE] it and create a 4D hypercube. 00:55:06.810 --> 00:55:08.120 align:middle line:90% So let's make it simple. 00:55:08.120 --> 00:55:10.400 align:middle line:84% Let's say the image was only 1D, just 00:55:10.400 --> 00:55:12.160 align:middle line:90% this part of the line here. 00:55:12.160 --> 00:55:15.752 align:middle line:84% If you take this frequency transform-- 00:55:15.752 --> 00:55:18.540 align:middle line:84% actually, [INAUDIBLE] transform looks like that. 00:55:18.540 --> 00:55:22.040 align:middle line:90% 00:55:22.040 --> 00:55:24.280 align:middle line:84% If you take this frequency transform, 00:55:24.280 --> 00:55:26.600 align:middle line:84% you will capture-- there will be some 1D frequency 00:55:26.600 --> 00:55:29.550 align:middle line:90% transform that looks like this. 00:55:29.550 --> 00:55:32.930 align:middle line:90% And what we're going to do is-- 00:55:32.930 --> 00:55:34.915 align:middle line:90% so let's go step by step. 00:55:34.915 --> 00:55:36.290 align:middle line:84% So the frequency domain-- this is 00:55:36.290 --> 00:55:38.540 align:middle line:90% a little bit of a [INAUDIBLE]. 00:55:38.540 --> 00:55:40.820 align:middle line:84% We'll come back and talk about 4D techniques 00:55:40.820 --> 00:55:42.380 align:middle line:90% in a couple of lectures. 00:55:42.380 --> 00:55:44.450 align:middle line:84% And this will become more clear at that time. 00:55:44.450 --> 00:55:46.370 align:middle line:84% But basically, we have some variation 00:55:46.370 --> 00:55:49.340 align:middle line:84% in x and some variation [INAUDIBLE].. 00:55:49.340 --> 00:55:50.520 align:middle line:90% We saw there. 00:55:50.520 --> 00:55:53.150 align:middle line:84% And that's because it was 100 by 9. 00:55:53.150 --> 00:55:56.375 align:middle line:90% In this case, it's 200 by 9. 00:55:56.375 --> 00:55:59.840 align:middle line:90% 00:55:59.840 --> 00:56:01.399 align:middle line:90% That's 200 [INAUDIBLE] actually. 00:56:01.399 --> 00:56:05.311 align:middle line:90% 00:56:05.311 --> 00:56:09.400 align:middle line:84% 200 by 9, we're just thinking [INAUDIBLE] data. 00:56:09.400 --> 00:56:13.630 align:middle line:84% And then our sensor, however, is just 1D in this case. 00:56:13.630 --> 00:56:17.770 align:middle line:84% So although we want to capture 200 here and nine here, 00:56:17.770 --> 00:56:19.850 align:middle line:84% we cannot capture that much from the sensor. 00:56:19.850 --> 00:56:23.350 align:middle line:84% So what we basically do is take that 1D sensor-- 00:56:23.350 --> 00:56:26.530 align:middle line:90% [INAUDIBLE] 00:56:26.530 --> 00:56:28.240 align:middle line:84% We're going to take that 1D sensor, 00:56:28.240 --> 00:56:32.500 align:middle line:84% and it's going to capture different parts of the signal. 00:56:32.500 --> 00:56:34.630 align:middle line:84% And then we're going to reshape it. 00:56:34.630 --> 00:56:36.220 align:middle line:90% We're going to chop-- 00:56:36.220 --> 00:56:37.670 align:middle line:84% we're going to chop this part over 00:56:37.670 --> 00:56:40.510 align:middle line:84% to here, chop this part over to here. 00:56:40.510 --> 00:56:41.890 align:middle line:84% This one stays in the same place. 00:56:41.890 --> 00:56:43.280 align:middle line:90% Take this part [INAUDIBLE] here. 00:56:43.280 --> 00:56:44.990 align:middle line:90% And this part, goes over here. 00:56:44.990 --> 00:56:50.103 align:middle line:84% And now, from a 1D signal, you have created a 2D length. 00:56:50.103 --> 00:56:52.020 align:middle line:84% And it's not really visually clear, I'm sorry, 00:56:52.020 --> 00:56:54.206 align:middle line:90% because [INAUDIBLE] scheme. 00:56:54.206 --> 00:56:57.470 align:middle line:90% [INAUDIBLE] 00:56:57.470 --> 00:57:00.040 align:middle line:90% And from that, we can recover-- 00:57:00.040 --> 00:57:01.790 align:middle line:84% AUDIENCE: So you have all the coefficients 00:57:01.790 --> 00:57:03.110 align:middle line:90% for the entire image. 00:57:03.110 --> 00:57:03.370 align:middle line:90% RAMESH RASKAR: Exactly. 00:57:03.370 --> 00:57:04.245 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:57:04.245 --> 00:57:06.526 align:middle line:84% RAMESH RASKAR: The whole coefficient [INAUDIBLE].. 00:57:06.526 --> 00:57:07.340 align:middle line:90% Yes? 00:57:07.340 --> 00:57:08.405 align:middle line:84% AUDIENCE: What about the [INAUDIBLE]?? 00:57:08.405 --> 00:57:08.960 align:middle line:90% RAMESH RASKAR: Sorry? 00:57:08.960 --> 00:57:10.370 align:middle line:90% AUDIENCE: The [? reason. ?] 00:57:10.370 --> 00:57:11.410 align:middle line:84% RAMESH RASKAR: Frequency domain [INAUDIBLE]?? 00:57:11.410 --> 00:57:11.810 align:middle line:90% AUDIENCE: Yes. 00:57:11.810 --> 00:57:13.400 align:middle line:84% RAMESH RASKAR: So we'll talk about that. 00:57:13.400 --> 00:57:15.150 align:middle line:84% If you think about this particular problem 00:57:15.150 --> 00:57:17.930 align:middle line:84% where we have overlap or undershoot, 00:57:17.930 --> 00:57:21.110 align:middle line:90% there's also a [INAUDIBLE]. 00:57:21.110 --> 00:57:23.720 align:middle line:84% Because we're trying to capture more signal, 00:57:23.720 --> 00:57:26.200 align:middle line:84% but we don't have enough bandwidth here. 00:57:26.200 --> 00:57:28.230 align:middle line:84% And the same exact problem here as well. 00:57:28.230 --> 00:57:31.370 align:middle line:84% So you;re going to assume that the scene doesn't have 00:57:31.370 --> 00:57:32.782 align:middle line:90% [INAUDIBLE]. 00:57:32.782 --> 00:57:36.867 align:middle line:84% So we're [INAUDIBLE] and must [INAUDIBLE].. 00:57:36.867 --> 00:57:38.450 align:middle line:84% AUDIENCE: But doesn't it mean that you 00:57:38.450 --> 00:57:41.832 align:middle line:84% lose [INAUDIBLE] data [INAUDIBLE] equipment? 00:57:41.832 --> 00:57:43.040 align:middle line:90% RAMESH RASKAR: You can only-- 00:57:43.040 --> 00:57:45.945 align:middle line:84% so you [INAUDIBLE] only 200 pixels in the world. 00:57:45.945 --> 00:57:48.070 align:middle line:84% AUDIENCE: Even if the spectrum is not [INAUDIBLE],, 00:57:48.070 --> 00:57:48.940 align:middle line:90% it's very simple. 00:57:48.940 --> 00:57:50.580 align:middle line:90% You need to sample [INAUDIBLE]. 00:57:50.580 --> 00:57:52.210 align:middle line:84% RAMESH RASKAR: You have to hope that-- 00:57:52.210 --> 00:57:55.580 align:middle line:84% you hope that your spectrum looks something like this. 00:57:55.580 --> 00:57:57.820 align:middle line:84% If your spectrum actually has lots of energy 00:57:57.820 --> 00:58:01.362 align:middle line:84% up here and down there, then, yes, you'd get [? RPS. ?] 00:58:01.362 --> 00:58:04.390 align:middle line:84% AUDIENCE: Even-- and even you can separate this signal 00:58:04.390 --> 00:58:06.925 align:middle line:84% with a much lower frequency than the spectrum [? specific ?] 00:58:06.925 --> 00:58:07.440 align:middle line:90% right? 00:58:07.440 --> 00:58:08.710 align:middle line:90% RAMESH RASKAR: Repeat that? 00:58:08.710 --> 00:58:12.430 align:middle line:84% AUDIENCE: You can separate in much lower frequency 00:58:12.430 --> 00:58:16.060 align:middle line:90% than actually signal separate? 00:58:16.060 --> 00:58:17.290 align:middle line:90% RAMESH RASKAR: Using what? 00:58:17.290 --> 00:58:18.415 align:middle line:90% Using how much [INAUDIBLE]? 00:58:18.415 --> 00:58:19.710 align:middle line:90% AUDIENCE: Use the tool. 00:58:19.710 --> 00:58:22.990 align:middle line:90% No, just simply low frequency. 00:58:22.990 --> 00:58:24.130 align:middle line:90% RAMESH RASKAR: Just take-- 00:58:24.130 --> 00:58:25.397 align:middle line:90% AUDIENCE: For this center one. 00:58:25.397 --> 00:58:26.230 align:middle line:90% RAMESH RASKAR: Yeah. 00:58:26.230 --> 00:58:28.150 align:middle line:84% That's exactly what happens in JPEG. 00:58:28.150 --> 00:58:30.880 align:middle line:84% You take whatever 4-megapixel image, 00:58:30.880 --> 00:58:32.770 align:middle line:84% and then you're able to represent it 00:58:32.770 --> 00:58:36.880 align:middle line:84% as a half-megapixel image, because you're able to take 00:58:36.880 --> 00:58:39.250 align:middle line:90% only the lower frequencies-- 00:58:39.250 --> 00:58:41.660 align:middle line:84% lower frequencies and represent them above. 00:58:41.660 --> 00:58:43.160 align:middle line:84% And you take the higher frequencies, 00:58:43.160 --> 00:58:44.868 align:middle line:84% but we don't represent them [INAUDIBLE].. 00:58:44.868 --> 00:58:49.288 align:middle line:84% That's how JPEG [INAUDIBLE] frequency [INAUDIBLE].. 00:58:49.288 --> 00:58:50.330 align:middle line:90% AUDIENCE: So you assume-- 00:58:50.330 --> 00:58:52.247 align:middle line:84% RAMESH RASKAR: But we cannot do that for real. 00:58:52.247 --> 00:58:53.370 align:middle line:90% We can do that in optics. 00:58:53.370 --> 00:58:54.320 align:middle line:90% AUDIENCE: Yes. 00:58:54.320 --> 00:58:56.028 align:middle line:84% RAMESH RASKAR: You can do it in software. 00:58:56.028 --> 00:58:58.190 align:middle line:90% So here's a method to do that-- 00:58:58.190 --> 00:59:01.670 align:middle line:84% not the [INAUDIBLE],, but the remapping in optics. 00:59:01.670 --> 00:59:03.772 align:middle line:84% AUDIENCE: So you still assume it's overcycle? 00:59:03.772 --> 00:59:04.980 align:middle line:90% RAMESH RASKAR: Yeah, exactly. 00:59:04.980 --> 00:59:06.980 align:middle line:84% So if the world actually had a checkerboard, 00:59:06.980 --> 00:59:09.230 align:middle line:84% an extremely high resolution checkerboard, 00:59:09.230 --> 00:59:10.560 align:middle line:90% then you'd have problems. 00:59:10.560 --> 00:59:11.540 align:middle line:90% It's a similar effect-- 00:59:11.540 --> 00:59:13.582 align:middle line:84% your software is where you really find structure. 00:59:13.582 --> 00:59:15.110 align:middle line:90% And you just take the picture. 00:59:15.110 --> 00:59:17.090 align:middle line:84% You see these areas and artifacts. 00:59:17.090 --> 00:59:19.220 align:middle line:84% Or, if you have a-- if you take a fence, 00:59:19.220 --> 00:59:22.490 align:middle line:84% we have [INAUDIBLE] take picture [INAUDIBLE].. 00:59:22.490 --> 00:59:25.078 align:middle line:90% So it's the same problem. 00:59:25.078 --> 00:59:30.140 align:middle line:84% It's just [INAUDIBLE] you cannot get your own [INAUDIBLE] unless 00:59:30.140 --> 00:59:32.360 align:middle line:84% you do them [? frame by frame. ?] I 00:59:32.360 --> 00:59:34.530 align:middle line:90% won't go into detail. 00:59:34.530 --> 00:59:37.100 align:middle line:84% AUDIENCE: Ramesh, do you have to apply this to each channel, 00:59:37.100 --> 00:59:38.120 align:middle line:90% the [INAUDIBLE]? 00:59:38.120 --> 00:59:38.480 align:middle line:90% RAMESH RASKAR: Yeah. 00:59:38.480 --> 00:59:40.230 align:middle line:84% You have to do it for every color channel. 00:59:40.230 --> 00:59:42.660 align:middle line:90% That's right. 00:59:42.660 --> 00:59:44.290 align:middle line:84% And from there, you have captured it. 00:59:44.290 --> 00:59:49.610 align:middle line:84% So let's look at the [INAUDIBLE] of how exactly this works. 00:59:49.610 --> 00:59:53.960 align:middle line:84% And there's a [INAUDIBLE] explanation that many of you 00:59:53.960 --> 00:59:55.610 align:middle line:90% have seen. 00:59:55.610 --> 01:00:00.440 align:middle line:84% And I'm going to give you a very simple, intuitive reasoning of 01:00:00.440 --> 01:00:02.850 align:middle line:90% [INAUDIBLE]. 01:00:02.850 --> 01:00:07.590 align:middle line:84% So let's come to this problem of replacing 01:00:07.590 --> 01:00:12.000 align:middle line:84% a pinhole with this new type of mask, 01:00:12.000 --> 01:00:14.720 align:middle line:90% which we call heterodyned mask. 01:00:14.720 --> 01:00:23.580 align:middle line:84% So here we have a pinhole and now this part again, a pinhole. 01:00:23.580 --> 01:00:25.050 align:middle line:84% And we have nine values coming in. 01:00:25.050 --> 01:00:27.840 align:middle line:90% 01:00:27.840 --> 01:00:29.884 align:middle line:90% [INAUDIBLE] minus 4. 01:00:29.884 --> 01:00:35.400 align:middle line:84% [INAUDIBLE] plus 4, 4, [INAUDIBLE].. 01:00:35.400 --> 01:00:37.271 align:middle line:84% And then we do the next [INAUDIBLE] over. 01:00:37.271 --> 01:00:40.030 align:middle line:90% 01:00:40.030 --> 01:00:41.780 align:middle line:84% And this one's doing the same thing again. 01:00:41.780 --> 01:00:50.450 align:middle line:90% 01:00:50.450 --> 01:00:53.112 align:middle line:84% And by using this trick, you're not 01:00:53.112 --> 01:00:54.320 align:middle line:90% going to get more resolution. 01:00:54.320 --> 01:00:55.490 align:middle line:90% You're still have some-- 01:00:55.490 --> 01:00:57.240 align:middle line:84% miss some information in the [INAUDIBLE].. 01:00:57.240 --> 01:00:59.070 align:middle line:84% It's just that we don't lose the light. 01:00:59.070 --> 01:01:03.410 align:middle line:84% We just lose those spatial frequencies. 01:01:03.410 --> 01:01:07.970 align:middle line:84% So small digression, and we're going to talk about something 01:01:07.970 --> 01:01:11.290 align:middle line:90% called [INAUDIBLE] multiplexing. 01:01:11.290 --> 01:01:17.786 align:middle line:90% 01:01:17.786 --> 01:01:22.950 align:middle line:84% And the problem is really easy to explain in this [INAUDIBLE].. 01:01:22.950 --> 01:01:26.340 align:middle line:90% Let's say I give you-- 01:01:26.340 --> 01:01:33.560 align:middle line:84% there are several bags with different weight, weight 1 01:01:33.560 --> 01:01:38.400 align:middle line:84% to weight 9, and I tell you to weigh them. 01:01:38.400 --> 01:01:41.390 align:middle line:84% And at the end, I want a solution 01:01:41.390 --> 01:01:46.290 align:middle line:84% of each-- what the weight of each of the bags is. 01:01:46.290 --> 01:01:50.870 align:middle line:84% [INAUDIBLE] the weighing scale, put each of the bags one 01:01:50.870 --> 01:01:53.993 align:middle line:84% at a time, and you have a solution. 01:01:53.993 --> 01:01:55.660 align:middle line:84% What are some other ways you can put it? 01:01:55.660 --> 01:02:03.490 align:middle line:90% 01:02:03.490 --> 01:02:08.010 align:middle line:84% If you don't want to do one at a time, what can you do? 01:02:08.010 --> 01:02:08.510 align:middle line:90% Sorry? 01:02:08.510 --> 01:02:09.552 align:middle line:90% AUDIENCE: Do all of them. 01:02:09.552 --> 01:02:10.802 align:middle line:90% RAMESH RASKAR: Do all of them? 01:02:10.802 --> 01:02:13.170 align:middle line:84% But that doesn't give you [INAUDIBLE] composing of nine. 01:02:13.170 --> 01:02:14.820 align:middle line:90% You must make nine measurements. 01:02:14.820 --> 01:02:16.170 align:middle line:90% I'll give you that. 01:02:16.170 --> 01:02:17.880 align:middle line:90% You must make nine measurements. 01:02:17.880 --> 01:02:22.290 align:middle line:84% One choice is, you just put one bag in. 01:02:22.290 --> 01:02:25.710 align:middle line:90% And you have nine [INAUDIBLE]. 01:02:25.710 --> 01:02:29.310 align:middle line:84% But let's say your scale actually-- 01:02:29.310 --> 01:02:34.570 align:middle line:84% so let's say the weight is from 0 to 100. 01:02:34.570 --> 01:02:37.840 align:middle line:84% I'm sorry, the weight itself is 0 to 10. 01:02:37.840 --> 01:02:41.145 align:middle line:84% But let's say your scale doesn't work very well 01:02:41.145 --> 01:02:46.900 align:middle line:84% in the first couple of [INAUDIBLE] statement. 01:02:46.900 --> 01:02:52.780 align:middle line:84% So the scale works pretty well when it's 01:02:52.780 --> 01:02:54.890 align:middle line:90% in the radiant [INAUDIBLE]. 01:02:54.890 --> 01:02:57.180 align:middle line:84% But it doesn't work so well when the weight is too low 01:02:57.180 --> 01:02:59.360 align:middle line:90% or weight is too high. 01:02:59.360 --> 01:03:02.620 align:middle line:84% So what you want to do is stay in the [INAUDIBLE].. 01:03:02.620 --> 01:03:03.980 align:middle line:90% And cameras work the same way. 01:03:03.980 --> 01:03:05.750 align:middle line:84% That's why I'm not doing this one. 01:03:05.750 --> 01:03:07.960 align:middle line:84% If the light is too low, if the light's too high, 01:03:07.960 --> 01:03:10.240 align:middle line:84% the camera cannot figure out where exactly it is. 01:03:10.240 --> 01:03:11.740 align:middle line:90% But if the light is between-- 01:03:11.740 --> 01:03:15.220 align:middle line:84% if the light is going from 0 to 55, then 01:03:15.220 --> 01:03:18.010 align:middle line:84% when [INAUDIBLE] to 200, the camera 01:03:18.010 --> 01:03:20.850 align:middle line:84% works nice in a straight line, straight line [INAUDIBLE].. 01:03:20.850 --> 01:03:22.230 align:middle line:84% When it's too dark or too bright, 01:03:22.230 --> 01:03:26.100 align:middle line:84% a camera cannot handle it as well. 01:03:26.100 --> 01:03:29.610 align:middle line:84% And this similar analogy [INAUDIBLE].. 01:03:29.610 --> 01:03:31.335 align:middle line:84% So one solution is to do one at a time. 01:03:31.335 --> 01:03:35.875 align:middle line:84% The other solution is I can put a group of them. 01:03:35.875 --> 01:03:37.750 align:middle line:84% Let's start with [INAUDIBLE] number of three. 01:03:37.750 --> 01:03:39.042 align:middle line:90% Let's say I have three of them. 01:03:39.042 --> 01:03:41.610 align:middle line:90% 01:03:41.610 --> 01:03:45.040 align:middle line:84% I could put one at a time or I could put two at a time. 01:03:45.040 --> 01:03:50.790 align:middle line:84% Plus, I can do w1 plus w2, then I'll do w2 plus w3. 01:03:50.790 --> 01:03:55.430 align:middle line:90% And I would do a w1 plus w2. 01:03:55.430 --> 01:03:57.380 align:middle line:84% In fact, there's three measurements. 01:03:57.380 --> 01:04:00.290 align:middle line:84% From the three measurements, I can figure out what 01:04:00.290 --> 01:04:02.990 align:middle line:90% each of these [INAUDIBLE] are. 01:04:02.990 --> 01:04:06.300 align:middle line:84% And you can actually write this down as a [INAUDIBLE] system. 01:04:06.300 --> 01:04:09.470 align:middle line:84% I will say that [INAUDIBLE] as well. 01:04:09.470 --> 01:04:13.090 align:middle line:90% 01:04:13.090 --> 01:04:15.070 align:middle line:84% So let's say when I put the first two, 01:04:15.070 --> 01:04:21.600 align:middle line:84% my measurement is on 1, 2, and this one, 3. 01:04:21.600 --> 01:04:24.510 align:middle line:84% And all we have done here is we have said, 01:04:24.510 --> 01:04:29.530 align:middle line:84% I'm making measurements-- and 1, and 2, and 3. 01:04:29.530 --> 01:04:31.030 align:middle line:84% What I would like to know, actually, 01:04:31.030 --> 01:04:35.484 align:middle line:90% is the weights-- w1, w2, w3. 01:04:35.484 --> 01:04:38.300 align:middle line:90% 01:04:38.300 --> 01:04:44.130 align:middle line:84% But the way I have measured it is the first one 01:04:44.130 --> 01:04:48.440 align:middle line:84% is w1 plus w2 and [INAUDIBLE] w3. 01:04:48.440 --> 01:04:53.392 align:middle line:84% Second one is w2 and w3 [INAUDIBLE] w1. 01:04:53.392 --> 01:04:59.490 align:middle line:84% And [INAUDIBLE] that's all [INAUDIBLE].. 01:04:59.490 --> 01:05:01.720 align:middle line:84% And we're just following the system. 01:05:01.720 --> 01:05:04.760 align:middle line:84% And it can tell you, from looking [INAUDIBLE].. 01:05:04.760 --> 01:05:07.590 align:middle line:90% 01:05:07.590 --> 01:05:13.920 align:middle line:84% And especially for this, [INAUDIBLE] purely inevitable. 01:05:13.920 --> 01:05:18.840 align:middle line:84% So for three, this seems like a very good solution, 01:05:18.840 --> 01:05:22.540 align:middle line:84% because now we are staying somewhere in the middle range. 01:05:22.540 --> 01:05:25.240 align:middle line:84% We are not at the bottom range of this weight. 01:05:25.240 --> 01:05:28.020 align:middle line:84% And if you put all three of them together at the same time, 01:05:28.020 --> 01:05:30.450 align:middle line:84% you might go very far over here, as well. 01:05:30.450 --> 01:05:36.590 align:middle line:84% [INAUDIBLE] because we did the sum of two [INAUDIBLE].. 01:05:36.590 --> 01:05:38.030 align:middle line:90% So this is very convenient. 01:05:38.030 --> 01:05:42.150 align:middle line:84% Another part of multiplexing is basically the same concept. 01:05:42.150 --> 01:05:46.790 align:middle line:84% Instead of three [INAUDIBLE] 9 or any such measurements, 01:05:46.790 --> 01:05:50.870 align:middle line:84% I'm going to basically take about half of them randomly. 01:05:50.870 --> 01:05:52.700 align:middle line:84% And I take about half of the bags, 01:05:52.700 --> 01:05:55.550 align:middle line:84% put them together, take a measurement. 01:05:55.550 --> 01:06:00.140 align:middle line:84% Then I will put them apart, again take some other half, 01:06:00.140 --> 01:06:03.140 align:middle line:84% and do our fun measurements, and so on. 01:06:03.140 --> 01:06:05.988 align:middle line:84% See, if I have nine of these, I will create a matrix 01:06:05.988 --> 01:06:07.280 align:middle line:90% that looks something like this. 01:06:07.280 --> 01:06:08.240 align:middle line:90% I will have-- 01:06:08.240 --> 01:06:12.990 align:middle line:90% 01:06:12.990 --> 01:06:18.170 align:middle line:84% I will measure my grades that we want to go [INAUDIBLE].. 01:06:18.170 --> 01:06:22.026 align:middle line:84% My measurements are [INAUDIBLE] 1 through the 9. 01:06:22.026 --> 01:06:25.640 align:middle line:90% 01:06:25.640 --> 01:06:27.862 align:middle line:90% And what will we have here? 01:06:27.862 --> 01:06:28.820 align:middle line:90% What's the [INAUDIBLE]? 01:06:28.820 --> 01:06:31.555 align:middle line:90% AUDIENCE: [INAUDIBLE] 01:06:31.555 --> 01:06:32.430 align:middle line:90% RAMESH RASKAR: Sorry? 01:06:32.430 --> 01:06:33.555 align:middle line:90% AUDIENCE: Nine [INAUDIBLE]. 01:06:33.555 --> 01:06:36.390 align:middle line:90% RAMESH RASKAR: Nine [INAUDIBLE]. 01:06:36.390 --> 01:06:39.990 align:middle line:84% And this way, we put down the numbers here, 01:06:39.990 --> 01:06:43.050 align:middle line:84% we've got to put some random numbers here. 01:06:43.050 --> 01:06:44.790 align:middle line:84% Instead of putting all the bags that 01:06:44.790 --> 01:06:48.010 align:middle line:84% [INAUDIBLE] all but [INAUDIBLE] part of this. 01:06:48.010 --> 01:06:49.110 align:middle line:90% Let's head back. 01:06:49.110 --> 01:06:51.000 align:middle line:84% If I just put one bag at a time, what 01:06:51.000 --> 01:06:53.764 align:middle line:90% will this mean [INAUDIBLE]? 01:06:53.764 --> 01:06:54.960 align:middle line:90% AUDIENCE: [INAUDIBLE] 01:06:54.960 --> 01:06:58.764 align:middle line:84% RAMESH RASKAR: It will just 1, 1, 1, And also the number 0. 01:06:58.764 --> 01:07:00.920 align:middle line:90% That's [INAUDIBLE]. 01:07:00.920 --> 01:07:02.920 align:middle line:84% And same thing will be going along the diagonal. 01:07:02.920 --> 01:07:10.210 align:middle line:84% [INAUDIBLE] out of the nine, I will have at least four or five 01:07:10.210 --> 01:07:17.188 align:middle line:84% [INAUDIBLE] placed within-- put randomly 101, 10-- 01:07:17.188 --> 01:07:20.032 align:middle line:90% 4, 5, 6, 7, 8, 9. 01:07:20.032 --> 01:07:21.366 align:middle line:90% So I put-- I don't know. 01:07:21.366 --> 01:07:23.746 align:middle line:90% 01, 02. 01:07:23.746 --> 01:07:25.650 align:middle line:90% That's one sequence. 01:07:25.650 --> 01:07:27.610 align:middle line:84% And then I'll take the next one-- 01:07:27.610 --> 01:07:30.450 align:middle line:90% 01, whatever's next. 01:07:30.450 --> 01:07:33.580 align:middle line:84% I have nine such [INAUDIBLE] sequences with either four 01:07:33.580 --> 01:07:36.160 align:middle line:90% or five of one [INAUDIBLE]. 01:07:36.160 --> 01:07:40.225 align:middle line:84% And this is how [INAUDIBLE] sequence as [INAUDIBLE].. 01:07:40.225 --> 01:07:42.670 align:middle line:84% So it's sort of taking one measurement at a time. 01:07:42.670 --> 01:07:47.230 align:middle line:84% But you will take a linear combination, in this case just 01:07:47.230 --> 01:07:50.380 align:middle line:90% a sum of our [INAUDIBLE]. 01:07:50.380 --> 01:07:52.480 align:middle line:84% And the benefit of that is that you're staying 01:07:52.480 --> 01:07:56.130 align:middle line:90% in a range that's [INAUDIBLE]. 01:07:56.130 --> 01:08:00.820 align:middle line:84% I used the analogy of this one with where we're going. 01:08:00.820 --> 01:08:04.420 align:middle line:84% Using a pinhole is like doing one measurement at a time. 01:08:04.420 --> 01:08:08.110 align:middle line:84% You're only measuring one ray at a time. 01:08:08.110 --> 01:08:13.000 align:middle line:84% Using a lens [INAUDIBLE] is like putting all the bags 01:08:13.000 --> 01:08:14.860 align:middle line:90% at the same time. 01:08:14.860 --> 01:08:19.025 align:middle line:84% And what we will do now is place a mask where only about half 01:08:19.025 --> 01:08:20.275 align:middle line:90% the values are coming through. 01:08:20.275 --> 01:08:23.890 align:middle line:90% So how many variables? 01:08:23.890 --> 01:08:27.359 align:middle line:84% So you're gonna be putting on half the input variables, 01:08:27.359 --> 01:08:28.667 align:middle line:90% and measure those. 01:08:28.667 --> 01:08:33.325 align:middle line:90% 01:08:33.325 --> 01:08:34.950 align:middle line:84% [INAUDIBLE] analogy, where it is going, 01:08:34.950 --> 01:08:37.950 align:middle line:84% and I'm able to predict what the variables will be. 01:08:37.950 --> 01:08:43.189 align:middle line:90% 01:08:43.189 --> 01:08:47.149 align:middle line:84% So let's focus on this one slab here. 01:08:47.149 --> 01:08:52.620 align:middle line:90% 01:08:52.620 --> 01:09:02.770 align:middle line:84% If you just focus on one slab here, light comes in 2 and 3. 01:09:02.770 --> 01:09:04.450 align:middle line:90% Instead of 9, we'll say, 3 01:09:04.450 --> 01:09:07.330 align:middle line:84% This one's just blocking this light here and here. 01:09:07.330 --> 01:09:10.990 align:middle line:90% 01:09:10.990 --> 01:09:14.240 align:middle line:84% What I will do is I'll block it from certain directions. 01:09:14.240 --> 01:09:18.207 align:middle line:90% So let's say I will make this-- 01:09:18.207 --> 01:09:19.999 align:middle line:84% actually, 14 doesn't look that interesting. 01:09:19.999 --> 01:09:21.899 align:middle line:90% So let's go to [INAUDIBLE]. 01:09:21.899 --> 01:09:24.210 align:middle line:90% I'll make it 1-1-1-0-1-0-0. 01:09:24.210 --> 01:09:33.350 align:middle line:90% 01:09:33.350 --> 01:09:36.710 align:middle line:90% [INAUDIBLE] in our half. 01:09:36.710 --> 01:09:40.309 align:middle line:84% What's going to happen is, for a certain part of the image-- 01:09:40.309 --> 01:09:43.834 align:middle line:84% if you look at the pixel here, I shoot the-- this is my lens. 01:09:43.834 --> 01:09:45.529 align:middle line:90% This is my pixel. 01:09:45.529 --> 01:09:47.850 align:middle line:84% If I shoot the ray here, it's blocked. 01:09:47.850 --> 01:09:50.810 align:middle line:84% But if I shoot the ray here, it goes through. 01:09:50.810 --> 01:09:53.320 align:middle line:84% If I shoot the ray here, it's again blocked. 01:09:53.320 --> 01:09:55.400 align:middle line:84% So what I'm getting at this pixel is not 01:09:55.400 --> 01:10:00.980 align:middle line:84% sum of all the rays or some image of [? conventionalist. ?] 01:10:00.980 --> 01:10:04.400 align:middle line:84% If I go to the next one, I get some other combination, 01:10:04.400 --> 01:10:08.270 align:middle line:84% because for this one this one was blocked, but for this one 01:10:08.270 --> 01:10:10.080 align:middle line:90% this goes through. 01:10:10.080 --> 01:10:12.000 align:middle line:90% Same here. 01:10:12.000 --> 01:10:14.190 align:middle line:84% But now this one's blocked, and so on. 01:10:14.190 --> 01:10:18.150 align:middle line:84% Because of the displacement, the combination that we're getting 01:10:18.150 --> 01:10:19.327 align:middle line:90% is evolutionary. 01:10:19.327 --> 01:10:21.587 align:middle line:90% 01:10:21.587 --> 01:10:23.670 align:middle line:84% And once we have done the [INAUDIBLE] combination, 01:10:23.670 --> 01:10:26.610 align:middle line:84% which is what we have here, about half the light 01:10:26.610 --> 01:10:28.230 align:middle line:90% goes through. 01:10:28.230 --> 01:10:30.685 align:middle line:84% But what we have seen here is the linear combination. 01:10:30.685 --> 01:10:31.810 align:middle line:90% We have these measurements. 01:10:31.810 --> 01:10:34.190 align:middle line:84% We don't have the original radiuses. 01:10:34.190 --> 01:10:39.820 align:middle line:84% Just simply calculating it will recover these intensities. 01:10:39.820 --> 01:10:41.150 align:middle line:90% And there you go. 01:10:41.150 --> 01:10:45.594 align:middle line:84% We're back to a traditional [INAUDIBLE].. 01:10:45.594 --> 01:10:48.890 align:middle line:84% So that's a very easy way of-- very intuitive way 01:10:48.890 --> 01:10:53.670 align:middle line:84% of thinking about how we can use [INAUDIBLE] multiplexing with 01:10:53.670 --> 01:10:55.846 align:middle line:90% [INAUDIBLE]. 01:10:55.846 --> 01:10:57.420 align:middle line:90% Again, the solution. 01:10:57.420 --> 01:10:59.230 align:middle line:90% So we get now half the light. 01:10:59.230 --> 01:11:03.629 align:middle line:84% [INAUDIBLE] but this lasts half in here, half here. 01:11:03.629 --> 01:11:07.500 align:middle line:84% In this case, we just have on half of the [INAUDIBLE] 01:11:07.500 --> 01:11:08.620 align:middle line:90% and [INAUDIBLE]. 01:11:08.620 --> 01:11:11.796 align:middle line:84% So we are [INAUDIBLE] for the masking and the actual camera 01:11:11.796 --> 01:11:12.870 align:middle line:90% is ready to go. 01:11:12.870 --> 01:11:16.620 align:middle line:84% Fortunately, you have to do a lot of computation 01:11:16.620 --> 01:11:19.660 align:middle line:84% on this equation for the whole image. 01:11:19.660 --> 01:11:22.410 align:middle line:90% Image is 16 megapixel. 01:11:22.410 --> 01:11:25.730 align:middle line:84% This matrix could be 16 million times 16 million 01:11:25.730 --> 01:11:28.616 align:middle line:84% if you do it in a brute force fashion. 01:11:28.616 --> 01:11:34.090 align:middle line:84% But of course, you just [INAUDIBLE] simplest conversion 01:11:34.090 --> 01:11:38.830 align:middle line:84% [INAUDIBLE] based on this that make 01:11:38.830 --> 01:11:40.490 align:middle line:90% it really, really important. 01:11:40.490 --> 01:11:42.880 align:middle line:84% So we won't go into inversion, but we 01:11:42.880 --> 01:11:46.630 align:middle line:84% want to stay in problem as [INAUDIBLE] would be and say 01:11:46.630 --> 01:11:48.880 align:middle line:90% that it's possible [INAUDIBLE]. 01:11:48.880 --> 01:11:49.380 align:middle line:90% Yes? 01:11:49.380 --> 01:11:51.176 align:middle line:84% AUDIENCE: Can you process other one 01:11:51.176 --> 01:11:54.318 align:middle line:90% or is something [INAUDIBLE]? 01:11:54.318 --> 01:11:56.110 align:middle line:84% RAMESH RASKAR: That's a very good question. 01:11:56.110 --> 01:11:58.260 align:middle line:84% So what I show you, I kind of cheated. 01:11:58.260 --> 01:12:07.940 align:middle line:84% I showed you a mask that looks like cosine. 01:12:07.940 --> 01:12:10.160 align:middle line:84% And what we're doing, it was certainly realized just 01:12:10.160 --> 01:12:15.350 align:middle line:84% in the last one year, is that [INAUDIBLE] and cosine 01:12:15.350 --> 01:12:17.640 align:middle line:90% is one and the same thing. 01:12:17.640 --> 01:12:21.920 align:middle line:84% So if you take a bunch of cosines-- 01:12:21.920 --> 01:12:23.720 align:middle line:84% and [INAUDIBLE] information, by the way. 01:12:23.720 --> 01:12:25.650 align:middle line:90% So that's why we have it here. 01:12:25.650 --> 01:12:28.440 align:middle line:84% So if you take cosines of different frequency-- 01:12:28.440 --> 01:12:30.550 align:middle line:84% so this is like taking [INAUDIBLE].. 01:12:30.550 --> 01:12:34.520 align:middle line:84% And you take a [INAUDIBLE] take positions on the frequencies. 01:12:34.520 --> 01:12:40.064 align:middle line:84% What we're doing is we're projecting on, again, 01:12:40.064 --> 01:12:43.330 align:middle line:84% different [? carriers ?] and so on. 01:12:43.330 --> 01:12:46.652 align:middle line:90% So if we do [INAUDIBLE]. 01:12:46.652 --> 01:12:57.010 align:middle line:90% 01:12:57.010 --> 01:13:02.530 align:middle line:84% If I do cosines, and then I'm going 01:13:02.530 --> 01:13:07.010 align:middle line:90% to make them sharper, sharper-- 01:13:07.010 --> 01:13:09.115 align:middle line:84% if I place all of them together, you 01:13:09.115 --> 01:13:10.990 align:middle line:84% start getting that [INAUDIBLE] in the center. 01:13:10.990 --> 01:13:12.830 align:middle line:90% And you get a bright spot. 01:13:12.830 --> 01:13:16.160 align:middle line:84% And away from it, you get a dark spot. 01:13:16.160 --> 01:13:18.020 align:middle line:84% The sum of all of this, as you can imagine, 01:13:18.020 --> 01:13:21.524 align:middle line:84% ends up being something like [INAUDIBLE] and then 01:13:21.524 --> 01:13:24.772 align:middle line:84% [INAUDIBLE] and then repeat some variation. 01:13:24.772 --> 01:13:27.100 align:middle line:90% So that's the part [INAUDIBLE]. 01:13:27.100 --> 01:13:30.220 align:middle line:84% And it turns out we can place these cosines in such a way 01:13:30.220 --> 01:13:32.490 align:middle line:84% so that we actually get a [INAUDIBLE].. 01:13:32.490 --> 01:13:34.320 align:middle line:84% And [INAUDIBLE] really takes a lot of work. 01:13:34.320 --> 01:13:37.060 align:middle line:84% We'll talk about it, about how you can actually 01:13:37.060 --> 01:13:38.444 align:middle line:90% get binary [INAUDIBLE]. 01:13:38.444 --> 01:13:42.320 align:middle line:90% 01:13:42.320 --> 01:13:46.390 align:middle line:84% And so something that [INAUDIBLE].. 01:13:46.390 --> 01:13:50.060 align:middle line:90% 01:13:50.060 --> 01:13:52.843 align:middle line:84% Because printing a binary mask has been measured. 01:13:52.843 --> 01:13:54.260 align:middle line:84% It's more convenient than printing 01:13:54.260 --> 01:13:58.730 align:middle line:84% a mask where [INAUDIBLE] is changing [INAUDIBLE].. 01:13:58.730 --> 01:14:01.990 align:middle line:84% AUDIENCE: And when you phrase it as a linear system, 01:14:01.990 --> 01:14:02.830 align:middle line:90% there is no-- 01:14:02.830 --> 01:14:05.380 align:middle line:84% I mean, no different, no sense in having-- 01:14:05.380 --> 01:14:06.940 align:middle line:90% blocking half the light or so. 01:14:06.940 --> 01:14:10.020 align:middle line:84% I mean, partially, with a partial transparency, 01:14:10.020 --> 01:14:11.822 align:middle line:84% but you could just go binary anyway. 01:14:11.822 --> 01:14:13.030 align:middle line:90% RAMESH RASKAR: Yeah, exactly. 01:14:13.030 --> 01:14:13.780 align:middle line:90% If you go binary-- 01:14:13.780 --> 01:14:15.040 align:middle line:90% AUDIENCE: [INAUDIBLE] 01:14:15.040 --> 01:14:16.150 align:middle line:84% RAMESH RASKAR: You're still going to lose half the light. 01:14:16.150 --> 01:14:16.750 align:middle line:90% AUDIENCE: Yeah, yeah. 01:14:16.750 --> 01:14:18.580 align:middle line:84% RAMESH RASKAR: [INAUDIBLE] some of this. 01:14:18.580 --> 01:14:19.205 align:middle line:90% AUDIENCE: Yeah. 01:14:19.205 --> 01:14:22.850 align:middle line:84% But I'm saying, instead of using [INAUDIBLE] and say 0.75 or so, 01:14:22.850 --> 01:14:23.660 align:middle line:90% I mean, [INAUDIBLE] 01:14:23.660 --> 01:14:25.160 align:middle line:84% RAMESH RASKAR: That's a great point. 01:14:25.160 --> 01:14:28.300 align:middle line:84% So let me rephrase what this fellow over here is saying. 01:14:28.300 --> 01:14:30.760 align:middle line:84% He's saying instead of ones and zeros, 01:14:30.760 --> 01:14:32.710 align:middle line:84% which means the sum of all this is still 01:14:32.710 --> 01:14:37.210 align:middle line:84% half of the total light, why not make this 1 and 0.75, 01:14:37.210 --> 01:14:39.160 align:middle line:90% and 1, 1, 1.75? 01:14:39.160 --> 01:14:40.960 align:middle line:84% So you still have this variation, 01:14:40.960 --> 01:14:43.360 align:middle line:84% but most of the light is going through. 01:14:43.360 --> 01:14:46.390 align:middle line:84% And it's actually a great topic for research as to what exactly 01:14:46.390 --> 01:14:47.770 align:middle line:90% this pattern should be. 01:14:47.770 --> 01:14:51.430 align:middle line:84% And for us, binary was very convenient on 0. 01:14:51.430 --> 01:14:53.770 align:middle line:84% And as you play with that parameter space, 01:14:53.770 --> 01:14:57.290 align:middle line:84% you get many, many different interesting solutions. 01:14:57.290 --> 01:15:00.192 align:middle line:84% Some of them are geared more towards getting to the photo, 01:15:00.192 --> 01:15:02.275 align:middle line:84% and some of them are geared more towards capturing 01:15:02.275 --> 01:15:03.610 align:middle line:90% the lightfield. 01:15:03.610 --> 01:15:06.079 align:middle line:84% So for example, if this is all ones, 01:15:06.079 --> 01:15:07.412 align:middle line:90% then you [INAUDIBLE] your photo. 01:15:07.412 --> 01:15:11.650 align:middle line:84% If this was all ones, that means every point is transferred, 01:15:11.650 --> 01:15:14.080 align:middle line:84% which means there's not much [INAUDIBLE].. 01:15:14.080 --> 01:15:17.770 align:middle line:84% If you put all zeros, that means it's [INAUDIBLE] go through. 01:15:17.770 --> 01:15:20.705 align:middle line:84% If you use a [? spectrum, ?] then it's a random [INAUDIBLE].. 01:15:20.705 --> 01:15:24.150 align:middle line:84% But if you use any [INAUDIBLE] in between, then [INAUDIBLE] 01:15:24.150 --> 01:15:26.428 align:middle line:90% optimizing [INAUDIBLE]. 01:15:26.428 --> 01:15:27.707 align:middle line:90% Yes? 01:15:27.707 --> 01:15:30.730 align:middle line:84% AUDIENCE: I guess one [INAUDIBLE] 01:15:30.730 --> 01:15:31.890 align:middle line:90% equations [INAUDIBLE]. 01:15:31.890 --> 01:15:35.620 align:middle line:90% 01:15:35.620 --> 01:15:37.370 align:middle line:90% Because holding [INAUDIBLE]. 01:15:37.370 --> 01:15:40.476 align:middle line:90% 01:15:40.476 --> 01:15:42.246 align:middle line:90% They say others [INAUDIBLE]. 01:15:42.246 --> 01:15:44.317 align:middle line:84% You [INAUDIBLE] in the Fourier [? trans-- ?] 01:15:44.317 --> 01:15:45.275 align:middle line:90% in Fourier [INAUDIBLE]. 01:15:45.275 --> 01:15:47.310 align:middle line:90% And you told me it's very easy. 01:15:47.310 --> 01:15:52.185 align:middle line:84% And it's very easy to figure out what kind of [INAUDIBLE].. 01:15:52.185 --> 01:15:53.060 align:middle line:90% RAMESH RASKAR: Right. 01:15:53.060 --> 01:15:53.660 align:middle line:90% Exactly. 01:15:53.660 --> 01:15:55.770 align:middle line:84% AUDIENCE: And I wonder what [INAUDIBLE] similar effect 01:15:55.770 --> 01:15:56.205 align:middle line:90% [INAUDIBLE]. 01:15:56.205 --> 01:15:57.747 align:middle line:84% RAMESH RASKAR: So I really would like 01:15:57.747 --> 01:16:00.390 align:middle line:84% to do [INAUDIBLE] optical domain, 01:16:00.390 --> 01:16:01.770 align:middle line:90% but we don't have the pleasure. 01:16:01.770 --> 01:16:03.960 align:middle line:84% The light is [INAUDIBLE].. through that light. 01:16:03.960 --> 01:16:06.790 align:middle line:84% And we can only do blocking and unblocking. 01:16:06.790 --> 01:16:11.050 align:middle line:84% So if you had a way to sense the image first, and mostly 01:16:11.050 --> 01:16:14.190 align:middle line:84% assess the image, and change your mask so that you 01:16:14.190 --> 01:16:17.220 align:middle line:84% can choose the right parameters for the scene, 01:16:17.220 --> 01:16:19.050 align:middle line:90% then what you're saying is true. 01:16:19.050 --> 01:16:21.900 align:middle line:90% So you get a softer compression. 01:16:21.900 --> 01:16:24.473 align:middle line:84% You get to look at the signal before you compress. 01:16:24.473 --> 01:16:27.620 align:middle line:84% AUDIENCE: But I thought you could [INAUDIBLE] it's not 01:16:27.620 --> 01:16:28.570 align:middle line:90% focused on the image. 01:16:28.570 --> 01:16:33.050 align:middle line:84% The image won't have any high level [INAUDIBLE].. 01:16:33.050 --> 01:16:38.542 align:middle line:84% But [INAUDIBLE] and you cannot capture frequencies like that. 01:16:38.542 --> 01:16:39.750 align:middle line:90% RAMESH RASKAR: So frequently. 01:16:39.750 --> 01:16:40.500 align:middle line:90% AUDIENCE: Exactly. 01:16:40.500 --> 01:16:41.868 align:middle line:90% And it doesn't [INAUDIBLE]. 01:16:41.868 --> 01:16:45.972 align:middle line:84% But [INAUDIBLE] we can [INAUDIBLE].. 01:16:45.972 --> 01:16:46.930 align:middle line:90% RAMESH RASKAR: Exactly. 01:16:46.930 --> 01:16:49.750 align:middle line:84% So what [? Deena ?] is just asking is, 01:16:49.750 --> 01:16:55.150 align:middle line:84% if I know that a machine is [INAUDIBLE] 200 pixels and 9 01:16:55.150 --> 01:16:58.450 align:middle line:84% variations-- so let's say for the photograph, what I'm going 01:16:58.450 --> 01:17:00.820 align:middle line:84% to do is put a photograph here and then 01:17:00.820 --> 01:17:02.230 align:middle line:90% capture its lightfield. 01:17:02.230 --> 01:17:05.170 align:middle line:84% And I'm going to [INAUDIBLE] 200 pixels. 01:17:05.170 --> 01:17:08.680 align:middle line:84% And I have nine views for each of those pixels. 01:17:08.680 --> 01:17:11.570 align:middle line:84% Then what I've done is built the correct system. 01:17:11.570 --> 01:17:13.720 align:middle line:84% But let's say somebody gave me a photograph which 01:17:13.720 --> 01:17:15.025 align:middle line:90% had, actually, 400 pixels. 01:17:15.025 --> 01:17:18.700 align:middle line:90% 01:17:18.700 --> 01:17:23.855 align:middle line:84% And, let's say [INAUDIBLE] 300 pixels, and then only six 01:17:23.855 --> 01:17:24.355 align:middle line:90% views. 01:17:24.355 --> 01:17:26.890 align:middle line:90% 01:17:26.890 --> 01:17:28.530 align:middle line:90% I don't know the priority. 01:17:28.530 --> 01:17:31.330 align:middle line:84% But you just note there is a 200-pixel photograph 01:17:31.330 --> 01:17:33.100 align:middle line:90% or a 300-pixel photograph. 01:17:33.100 --> 01:17:36.760 align:middle line:84% I will continue to decode that as a 200-pixel photo. 01:17:36.760 --> 01:17:40.090 align:middle line:84% But the problem will be, because this team has higher 01:17:40.090 --> 01:17:41.770 align:middle line:84% frequencies, different areas, it's 01:17:41.770 --> 01:17:43.120 align:middle line:90% the same question you're asking. 01:17:43.120 --> 01:17:45.130 align:middle line:84% And you have no idea of knowing [INAUDIBLE].. 01:17:45.130 --> 01:17:49.390 align:middle line:84% So a typical solution in any signal processing, 01:17:49.390 --> 01:17:53.140 align:middle line:84% any device that has been sampling, is you prefigure. 01:17:53.140 --> 01:17:56.560 align:middle line:84% You reduce [INAUDIBLE] by [INAUDIBLE] or cutting off 01:17:56.560 --> 01:17:59.620 align:middle line:84% high frequencies so that it will not [INAUDIBLE] 01:17:59.620 --> 01:18:01.260 align:middle line:90% as low frequencies. 01:18:01.260 --> 01:18:03.010 align:middle line:84% In the optical domain, unfortunately, it's 01:18:03.010 --> 01:18:04.450 align:middle line:90% very difficult to do. 01:18:04.450 --> 01:18:07.780 align:middle line:84% [INAUDIBLE] that will convert something 01:18:07.780 --> 01:18:10.350 align:middle line:84% that's 300 pixels to 200 pixels purely optics. 01:18:10.350 --> 01:18:12.070 align:middle line:90% You can do the software. 01:18:12.070 --> 01:18:15.220 align:middle line:84% You can [INAUDIBLE] or you can smooth [INAUDIBLE] software. 01:18:15.220 --> 01:18:18.280 align:middle line:84% But it's not that easy to do in [INAUDIBLE].. 01:18:18.280 --> 01:18:19.780 align:middle line:84% At least we don't have a solution. 01:18:19.780 --> 01:18:22.860 align:middle line:84% Maybe you will come up with a solution. 01:18:22.860 --> 01:18:26.482 align:middle line:84% And once you do that, then you can [INAUDIBLE].. 01:18:26.482 --> 01:18:30.860 align:middle line:84% AUDIENCE: So it says here that if [INAUDIBLE].. 01:18:30.860 --> 01:18:36.384 align:middle line:90% 01:18:36.384 --> 01:18:39.850 align:middle line:84% And so I'm thinking just about will happen to-- 01:18:39.850 --> 01:18:44.880 align:middle line:84% [INAUDIBLE] and the-- it's a complex number. 01:18:44.880 --> 01:18:49.020 align:middle line:84% And you have [INAUDIBLE] and the magnitude. 01:18:49.020 --> 01:18:51.520 align:middle line:84% Are you capturing both, or are you only capturing magnitude? 01:18:51.520 --> 01:18:52.770 align:middle line:90% RAMESH RASKAR: Only magnitude. 01:18:52.770 --> 01:18:54.750 align:middle line:84% So the other way to think about that 01:18:54.750 --> 01:18:58.230 align:middle line:84% is, as I was drawing these cosines, instead of one, zero, 01:18:58.230 --> 01:19:00.090 align:middle line:84% and [INAUDIBLE],, what you're doing 01:19:00.090 --> 01:19:01.540 align:middle line:90% is really truly a prediction. 01:19:01.540 --> 01:19:06.060 align:middle line:84% So the very first frequency we have is first cosine. 01:19:06.060 --> 01:19:09.080 align:middle line:90% 01:19:09.080 --> 01:19:12.350 align:middle line:84% The second one we have is the highest [INAUDIBLE].. 01:19:12.350 --> 01:19:14.720 align:middle line:84% The third one is that you will have the lowest cosine. 01:19:14.720 --> 01:19:18.095 align:middle line:84% And we have these nine cosines with four 01:19:18.095 --> 01:19:23.420 align:middle line:84% with plus, negative phase, and other four with [INAUDIBLE].. 01:19:23.420 --> 01:19:25.760 align:middle line:84% And the middle one is just [INAUDIBLE].. 01:19:25.760 --> 01:19:29.210 align:middle line:84% That's why I [INAUDIBLE] this particular pattern. 01:19:29.210 --> 01:19:31.318 align:middle line:84% And it turns out, instead of using cosines, 01:19:31.318 --> 01:19:32.735 align:middle line:84% we can do the binary [INAUDIBLE].. 01:19:32.735 --> 01:19:33.110 align:middle line:90% AUDIENCE: [INAUDIBLE] 01:19:33.110 --> 01:19:35.068 align:middle line:84% RAMESH RASKAR: That's the keyword [INAUDIBLE].. 01:19:35.068 --> 01:19:40.220 align:middle line:84% AUDIENCE: [INAUDIBLE] because now we [INAUDIBLE].. 01:19:40.220 --> 01:19:42.830 align:middle line:84% And are all these Fourier transformed different? 01:19:42.830 --> 01:19:44.853 align:middle line:90% Or are some of them [INAUDIBLE]? 01:19:44.853 --> 01:19:45.770 align:middle line:90% RAMESH RASKAR: No, no. 01:19:45.770 --> 01:19:47.797 align:middle line:84% They're very completely different. 01:19:47.797 --> 01:19:49.130 align:middle line:90% They're going to be [INAUDIBLE]. 01:19:49.130 --> 01:19:50.172 align:middle line:90% AUDIENCE: So [INAUDIBLE]? 01:19:50.172 --> 01:19:52.297 align:middle line:84% RAMESH RASKAR: Because, remember, I'm taking cosine 01:19:52.297 --> 01:19:53.475 align:middle line:90% and it's [INAUDIBLE]. 01:19:53.475 --> 01:19:57.820 align:middle line:90% 01:19:57.820 --> 01:20:01.790 align:middle line:84% So basically, again, I have nine unknowns and nine measurements. 01:20:01.790 --> 01:20:03.790 align:middle line:84% And I can't come up with any of these signatures 01:20:03.790 --> 01:20:05.680 align:middle line:90% as [INAUDIBLE]. 01:20:05.680 --> 01:20:07.710 align:middle line:84% I can mix them, bunch of ones and zeros. 01:20:07.710 --> 01:20:09.500 align:middle line:90% Just [INAUDIBLE] missing. 01:20:09.500 --> 01:20:12.932 align:middle line:84% Or I can do [INAUDIBLE] some kind of [INAUDIBLE] prediction. 01:20:12.932 --> 01:20:14.490 align:middle line:90% I can do [INAUDIBLE]. 01:20:14.490 --> 01:20:18.970 align:middle line:90% 01:20:18.970 --> 01:20:20.470 align:middle line:84% AUDIENCE: From the way you described 01:20:20.470 --> 01:20:24.020 align:middle line:84% putting the mask there, the way this matrix is formed, 01:20:24.020 --> 01:20:26.350 align:middle line:84% it seems like the second row is probably 01:20:26.350 --> 01:20:27.940 align:middle line:90% a shift of the first row. 01:20:27.940 --> 01:20:29.982 align:middle line:84% RAMESH RASKAR: It could be, because [INAUDIBLE].. 01:20:29.982 --> 01:20:30.934 align:middle line:90% So that's the [INAUDIBLE]. 01:20:30.934 --> 01:20:32.934 align:middle line:84% AUDIENCE: Because invariably, inversion probably 01:20:32.934 --> 01:20:36.010 align:middle line:84% can be easier, because you know the pattern of the matrix. 01:20:36.010 --> 01:20:37.660 align:middle line:84% And it's not a brute force inversion. 01:20:37.660 --> 01:20:38.665 align:middle line:84% Maybe you can do [? automation. ?] 01:20:38.665 --> 01:20:39.640 align:middle line:90% RAMESH RASKAR: Exactly. 01:20:39.640 --> 01:20:43.540 align:middle line:84% So all the beauty and elegance of the effort 01:20:43.540 --> 01:20:48.130 align:middle line:84% comes in in choosing the right type of mask and using-- 01:20:48.130 --> 01:20:51.130 align:middle line:84% codesigning a decoding scheme that's-- 01:20:51.130 --> 01:20:53.914 align:middle line:84% it doesn't amplify the noise, and so on. 01:20:53.914 --> 01:20:56.250 align:middle line:90% You had a question? 01:20:56.250 --> 01:20:59.130 align:middle line:84% AUDIENCE: I was going to ask [INAUDIBLE].. 01:20:59.130 --> 01:21:05.145 align:middle line:90% 01:21:05.145 --> 01:21:06.770 align:middle line:84% RAMESH RASKAR: That's the same question 01:21:06.770 --> 01:21:07.940 align:middle line:90% that [? Deena ?] is asking. 01:21:07.940 --> 01:21:11.710 align:middle line:84% In fact, I'll-- anything beyond 200, so the-- 01:21:11.710 --> 01:21:14.780 align:middle line:84% from 201 to 300 doesn't dominate the picture. 01:21:14.780 --> 01:21:16.460 align:middle line:84% I like to think of it as script domain 01:21:16.460 --> 01:21:23.144 align:middle line:84% because [INAUDIBLE] and because [INAUDIBLE].. 01:21:23.144 --> 01:21:25.102 align:middle line:84% AUDIENCE: [INAUDIBLE] from a little while back. 01:21:25.102 --> 01:21:33.180 align:middle line:84% Can't we use something with [INAUDIBLE] arrays [INAUDIBLE]?? 01:21:33.180 --> 01:21:35.580 align:middle line:84% RAMESH RASKAR: So it turns out [INAUDIBLE] 01:21:35.580 --> 01:21:38.430 align:middle line:84% whether you think about them as 4D real space 01:21:38.430 --> 01:21:41.820 align:middle line:84% or you think of light as wavelengths [INAUDIBLE] 01:21:41.820 --> 01:21:44.742 align:middle line:84% for on measuring the [INAUDIBLE].. 01:21:44.742 --> 01:21:46.200 align:middle line:84% AUDIENCE: Yeah, I was just thinking 01:21:46.200 --> 01:21:48.200 align:middle line:90% I could guide [INAUDIBLE]. 01:21:48.200 --> 01:21:50.150 align:middle line:90% 01:21:50.150 --> 01:21:51.900 align:middle line:84% RAMESH RASKAR: [INAUDIBLE] down from UCLA? 01:21:51.900 --> 01:21:52.483 align:middle line:90% AUDIENCE: Yes. 01:21:52.483 --> 01:21:53.070 align:middle line:90% [INAUDIBLE] 01:21:53.070 --> 01:21:55.375 align:middle line:90% 01:21:55.375 --> 01:21:56.250 align:middle line:90% RAMESH RASKAR: Right. 01:21:56.250 --> 01:21:57.820 align:middle line:84% But he's using diffraction for that. 01:21:57.820 --> 01:21:59.487 align:middle line:84% And we'll come back and talk about this. 01:21:59.487 --> 01:22:01.210 align:middle line:84% And we'll talk about [INAUDIBLE].. 01:22:01.210 --> 01:22:05.095 align:middle line:84% So a lot of you are wondering, where does this all fit in? 01:22:05.095 --> 01:22:08.260 align:middle line:84% This seems like a lot of math, a lot of algebra, 01:22:08.260 --> 01:22:10.810 align:middle line:84% or something you could just put a pinhole in and be 01:22:10.810 --> 01:22:12.590 align:middle line:90% done with it [INAUDIBLE]. 01:22:12.590 --> 01:22:15.850 align:middle line:84% So for example, right here, I am going 01:22:15.850 --> 01:22:19.210 align:middle line:84% to explain to you how the same exact technique can 01:22:19.210 --> 01:22:22.010 align:middle line:84% be manipulated in very different ways in two very [INAUDIBLE] 01:22:22.010 --> 01:22:23.990 align:middle line:90% series [INAUDIBLE]. 01:22:23.990 --> 01:22:27.340 align:middle line:84% AUDIENCE: So just a quick question for that. 01:22:27.340 --> 01:22:29.770 align:middle line:84% So have you already compared the results 01:22:29.770 --> 01:22:32.548 align:middle line:84% using this cosine mask with the binary ones? 01:22:32.548 --> 01:22:33.340 align:middle line:90% RAMESH RASKAR: Yes. 01:22:33.340 --> 01:22:33.840 align:middle line:90% Yes. 01:22:33.840 --> 01:22:35.290 align:middle line:90% In fact, here, let me show you. 01:22:35.290 --> 01:22:37.820 align:middle line:84% AUDIENCE: Because in theory, they should be equivalent. 01:22:37.820 --> 01:22:40.900 align:middle line:84% But in practice, because maybe some high frequencies 01:22:40.900 --> 01:22:43.780 align:middle line:84% that you introduce by the binary mask coming in-- 01:22:43.780 --> 01:22:46.270 align:middle line:84% and then you have to truncate at some point. 01:22:46.270 --> 01:22:48.355 align:middle line:84% RAMESH RASKAR: In simulation, they're identical. 01:22:48.355 --> 01:22:48.980 align:middle line:90% AUDIENCE: Yeah. 01:22:48.980 --> 01:22:50.855 align:middle line:84% RAMESH RASKAR: But you're right, because what 01:22:50.855 --> 01:22:56.500 align:middle line:84% isn't happening is when you print these cosine masks, 01:22:56.500 --> 01:22:59.920 align:middle line:84% your intensities are actually not transposed [INAUDIBLE] 01:22:59.920 --> 01:23:01.340 align:middle line:90% very well in cosine. 01:23:01.340 --> 01:23:06.364 align:middle line:84% But it's actually wearing something like a step function 01:23:06.364 --> 01:23:07.280 align:middle line:90% and so on. 01:23:07.280 --> 01:23:10.940 align:middle line:84% Because this digital levels by completing this [INAUDIBLE].. 01:23:10.940 --> 01:23:13.340 align:middle line:84% So that ended up creating a lot of trouble. 01:23:13.340 --> 01:23:14.773 align:middle line:84% AUDIENCE: And for the binary one, 01:23:14.773 --> 01:23:16.190 align:middle line:84% as well, because then you're going 01:23:16.190 --> 01:23:19.010 align:middle line:84% to have some sharp transitions that, in the Fourier domain, 01:23:19.010 --> 01:23:20.782 align:middle line:84% they may go all the way to high frequency. 01:23:20.782 --> 01:23:21.740 align:middle line:90% RAMESH RASKAR: Exactly. 01:23:21.740 --> 01:23:24.200 align:middle line:84% It's a conditioning of the [INAUDIBLE] transitions 01:23:24.200 --> 01:23:24.920 align:middle line:90% not right. 01:23:24.920 --> 01:23:27.730 align:middle line:84% And your model is incorrect, [INAUDIBLE].. 01:23:27.730 --> 01:23:29.970 align:middle line:90% So the calibration is very key. 01:23:29.970 --> 01:23:33.230 align:middle line:84% But the one benefit is that because now we're 01:23:33.230 --> 01:23:37.090 align:middle line:84% using masks rather than [INAUDIBLE],, 01:23:37.090 --> 01:23:39.640 align:middle line:84% you don't have to have the lens [INAUDIBLE] 01:23:39.640 --> 01:23:44.440 align:middle line:84% so that image of the lens is created as a sharply 01:23:44.440 --> 01:23:48.496 align:middle line:90% focused version on the sensor. 01:23:48.496 --> 01:23:51.520 align:middle line:84% In other words, very, very, strong constraint 01:23:51.520 --> 01:23:53.410 align:middle line:84% that we must put distance [INAUDIBLE] 01:23:53.410 --> 01:23:57.620 align:middle line:84% so that the image of the lens is formed on the sensor. 01:23:57.620 --> 01:23:59.510 align:middle line:90% We don't have that anymore. 01:23:59.510 --> 01:24:01.220 align:middle line:84% Even if we have some [INAUDIBLE],, 01:24:01.220 --> 01:24:03.170 align:middle line:84% there's some [INAUDIBLE],, some tolerance 01:24:03.170 --> 01:24:05.510 align:middle line:90% in [INAUDIBLE] of the mask. 01:24:05.510 --> 01:24:08.120 align:middle line:84% Because all we're doing is we're taking the values 01:24:08.120 --> 01:24:09.240 align:middle line:90% we want to measure. 01:24:09.240 --> 01:24:12.380 align:middle line:84% We're going to mix them around and get new measurements 01:24:12.380 --> 01:24:13.960 align:middle line:90% and [INAUDIBLE]. 01:24:13.960 --> 01:24:15.110 align:middle line:90% That gives us our freedom. 01:24:15.110 --> 01:24:20.620 align:middle line:84% So if you go back to the analogy of nine bags, it's rough-- 01:24:20.620 --> 01:24:23.540 align:middle line:90% 01:24:23.540 --> 01:24:26.560 align:middle line:84% it's roughly a very specific combination of those nine bags. 01:24:26.560 --> 01:24:30.010 align:middle line:84% It will be a slightly different combination of the nine bags. 01:24:30.010 --> 01:24:33.310 align:middle line:84% My mixing matrix [INAUDIBLE] most efficiently, 01:24:33.310 --> 01:24:36.790 align:middle line:84% but it would be good enough to do a reasonable job 01:24:36.790 --> 01:24:39.220 align:middle line:90% of [INAUDIBLE]. 01:24:39.220 --> 01:24:43.955 align:middle line:84% So that gives you a lot of flexibility [INAUDIBLE].. 01:24:43.955 --> 01:24:45.580 align:middle line:84% So now I'm going to show you how we can 01:24:45.580 --> 01:24:47.373 align:middle line:90% use this [INAUDIBLE] technique. 01:24:47.373 --> 01:24:48.790 align:middle line:84% AUDIENCE: So, Ramesh, but it seems 01:24:48.790 --> 01:24:51.273 align:middle line:84% that the binary mask made to do some diffraction 01:24:51.273 --> 01:24:51.940 align:middle line:90% artifact, right? 01:24:51.940 --> 01:24:52.330 align:middle line:90% RAMESH RASKAR: They do. 01:24:52.330 --> 01:24:53.290 align:middle line:90% They do. 01:24:53.290 --> 01:24:55.440 align:middle line:84% Are you going to talk about that? 01:24:55.440 --> 01:24:57.190 align:middle line:84% GUEST SPEAKER: I'm not sure I got to that. 01:24:57.190 --> 01:24:57.940 align:middle line:90% That's fine. 01:24:57.940 --> 01:24:59.440 align:middle line:84% You can see the results, and then we 01:24:59.440 --> 01:24:59.890 align:middle line:90% can talk about [INAUDIBLE]. 01:24:59.890 --> 01:25:00.280 align:middle line:90% RAMESH RASKAR: Yeah. 01:25:00.280 --> 01:25:02.530 align:middle line:84% And, just-- you can just go slow, because we just 01:25:02.530 --> 01:25:03.940 align:middle line:90% talked about it [INAUDIBLE]. 01:25:03.940 --> 01:25:04.690 align:middle line:90% GUEST SPEAKER: OK. 01:25:04.690 --> 01:25:06.590 align:middle line:90% So I--