WEBVTT 00:00:00.000 --> 00:00:02.460 align:middle line:84% The following content is provided under a Creative 00:00:02.460 --> 00:00:03.700 align:middle line:90% Commons license. 00:00:03.700 --> 00:00:06.090 align:middle line:84% Your support will help MIT OpenCourseWare 00:00:06.090 --> 00:00:10.060 align:middle line:84% continue to offer high-quality educational resources for free. 00:00:10.060 --> 00:00:12.690 align:middle line:84% To make a donation or to view additional materials 00:00:12.690 --> 00:00:16.620 align:middle line:84% from hundreds of MIT courses, visit MIT OpenCourseWare 00:00:16.620 --> 00:00:17.950 align:middle line:90% at ocw.mit.edu. 00:00:17.950 --> 00:00:21.128 align:middle line:90% 00:00:21.128 --> 00:00:22.795 align:middle line:84% PROFESSOR: All right, let's get started. 00:00:22.795 --> 00:00:27.140 align:middle line:90% 00:00:27.140 --> 00:00:29.380 align:middle line:84% So I think what's interesting about light 00:00:29.380 --> 00:00:33.472 align:middle line:84% is that it's used in so many interesting ways, 00:00:33.472 --> 00:00:35.680 align:middle line:84% whether it's programmed, whether it's not programmed, 00:00:35.680 --> 00:00:37.780 align:middle line:84% how it interacts with the world, [INAUDIBLE] 00:00:37.780 --> 00:00:41.380 align:middle line:84% there's direct bounce, multiple bounces, different wavelength, 00:00:41.380 --> 00:00:44.210 align:middle line:84% modulation, time, and space, it's a lot of fun. 00:00:44.210 --> 00:00:47.810 align:middle line:84% Like, for example, do you know how a TV remote control works? 00:00:47.810 --> 00:00:49.025 align:middle line:90% AUDIENCE: IR pulses. 00:00:49.025 --> 00:00:49.900 align:middle line:90% PROFESSOR: IR pulses. 00:00:49.900 --> 00:00:53.440 align:middle line:84% It's mostly optical, the output of RF-- 00:00:53.440 --> 00:01:00.280 align:middle line:84% but the LED of the remote is sending a code, 00:01:00.280 --> 00:01:02.590 align:middle line:84% optical code basically-- thanks-- 00:01:02.590 --> 00:01:06.323 align:middle line:84% over time to the photosensor on the TV. 00:01:06.323 --> 00:01:07.990 align:middle line:84% Now, why does it work in broad daylight? 00:01:07.990 --> 00:01:12.790 align:middle line:90% 00:01:12.790 --> 00:01:14.240 align:middle line:90% AUDIENCE: [INAUDIBLE] spectrum. 00:01:14.240 --> 00:01:15.782 align:middle line:84% PROFESSOR: It's a different spectrum. 00:01:15.782 --> 00:01:17.440 align:middle line:90% That's one benefit. 00:01:17.440 --> 00:01:18.350 align:middle line:90% That's not enough. 00:01:18.350 --> 00:01:21.095 align:middle line:84% AUDIENCE: Just looking at the differences between light, 00:01:21.095 --> 00:01:21.595 align:middle line:90% like peaks-- 00:01:21.595 --> 00:01:22.567 align:middle line:90% [INTERPOSING VOICES] 00:01:22.567 --> 00:01:23.400 align:middle line:90% AUDIENCE: --actual-- 00:01:23.400 --> 00:01:26.650 align:middle line:84% PROFESSOR: [LAUGHS] Actually, you can shine on the ceiling, 00:01:26.650 --> 00:01:29.365 align:middle line:90% and this will work. 00:01:29.365 --> 00:01:31.023 align:middle line:84% AUDIENCE: Just the variation in time. 00:01:31.023 --> 00:01:31.940 align:middle line:90% AUDIENCE: It has time. 00:01:31.940 --> 00:01:32.773 align:middle line:90% So it has the last-- 00:01:32.773 --> 00:01:34.815 align:middle line:84% AUDIENCE: Yeah, looking at peaks [? currently. ?] 00:01:34.815 --> 00:01:35.648 align:middle line:90% [INTERPOSING VOICES] 00:01:35.648 --> 00:01:36.940 align:middle line:90% PROFESSOR: Sorry, the last one? 00:01:36.940 --> 00:01:39.370 align:middle line:84% AUDIENCE: It has less energy than [INAUDIBLE] difference 00:01:39.370 --> 00:01:40.990 align:middle line:90% of pulse or something. 00:01:40.990 --> 00:01:41.740 align:middle line:90% PROFESSOR: Almost. 00:01:41.740 --> 00:01:43.052 align:middle line:90% What else [INAUDIBLE]? 00:01:43.052 --> 00:01:44.094 align:middle line:90% AUDIENCE: The modulation. 00:01:44.094 --> 00:01:44.860 align:middle line:90% AUDIENCE: Then the filter-- 00:01:44.860 --> 00:01:46.235 align:middle line:84% PROFESSOR: It's using modulation. 00:01:46.235 --> 00:01:49.510 align:middle line:84% So it's actually running at 40 kilohertz. 00:01:49.510 --> 00:01:52.240 align:middle line:84% So when it's 1, you're sending something 40 kilohertz, 00:01:52.240 --> 00:01:54.290 align:middle line:84% when it's 0, it's not sending anything. 00:01:54.290 --> 00:01:59.860 align:middle line:84% And so the AC component, which the carrier, is 40 kilohertz. 00:01:59.860 --> 00:02:02.620 align:middle line:90% And then the signal is 1 or 0. 00:02:02.620 --> 00:02:05.910 align:middle line:84% So it can record in presence of ambient light, 00:02:05.910 --> 00:02:07.510 align:middle line:84% because ambient light is mostly DC. 00:02:07.510 --> 00:02:11.410 align:middle line:84% AUDIENCE: But it can't-- well, so yesterday, 00:02:11.410 --> 00:02:13.990 align:middle line:84% I was watching TV, and all of a sudden my remote-- 00:02:13.990 --> 00:02:15.500 align:middle line:84% or the cable box stopped working. 00:02:15.500 --> 00:02:16.490 align:middle line:84% And I thought it was [INAUDIBLE] one of those, 00:02:16.490 --> 00:02:19.000 align:middle line:84% and there was in the [? wrong mode ?] or-- nothing 00:02:19.000 --> 00:02:19.840 align:middle line:90% worked. 00:02:19.840 --> 00:02:22.290 align:middle line:84% And then I noticed the TV light was blinking. 00:02:22.290 --> 00:02:24.790 align:middle line:84% And noticed-- I was sitting on the other-- at the TV remote, 00:02:24.790 --> 00:02:26.916 align:middle line:84% and it was just blasting IR light. 00:02:26.916 --> 00:02:29.660 align:middle line:84% But the cable box was just ignoring it, 00:02:29.660 --> 00:02:31.065 align:middle line:90% like it couldn't figure out-- 00:02:31.065 --> 00:02:33.190 align:middle line:84% PROFESSOR: Because you were pressing too many keys. 00:02:33.190 --> 00:02:35.500 align:middle line:84% AUDIENCE: I was holding one button on the other remote 00:02:35.500 --> 00:02:36.610 align:middle line:90% without knowing it. 00:02:36.610 --> 00:02:36.790 align:middle line:90% And-- 00:02:36.790 --> 00:02:38.080 align:middle line:84% PROFESSOR: So two different remotes 00:02:38.080 --> 00:02:39.220 align:middle line:84% were conflicting with each other. 00:02:39.220 --> 00:02:40.000 align:middle line:90% AUDIENCE: --remotes, yes. 00:02:40.000 --> 00:02:40.960 align:middle line:90% PROFESSOR: Exactly. 00:02:40.960 --> 00:02:44.170 align:middle line:84% So it's just a simple principle that we always 00:02:44.170 --> 00:02:46.480 align:middle line:90% associate with the TV remote. 00:02:46.480 --> 00:02:51.550 align:middle line:84% But can that also be used for photography or imaging? 00:02:51.550 --> 00:02:53.500 align:middle line:90% So the signal peaks-- 00:02:53.500 --> 00:02:57.220 align:middle line:84% the photodetector on the TV is decoding the signal. 00:02:57.220 --> 00:03:01.450 align:middle line:84% But that's basically a single pixel. 00:03:01.450 --> 00:03:03.760 align:middle line:84% Imagine if every pixel in the camera 00:03:03.760 --> 00:03:07.840 align:middle line:84% was made out of that photodetector that's decoding 00:03:07.840 --> 00:03:08.890 align:middle line:90% the 40-kilohertz signal. 00:03:08.890 --> 00:03:13.160 align:middle line:90% 00:03:13.160 --> 00:03:13.885 align:middle line:90% OK. 00:03:13.885 --> 00:03:15.650 align:middle line:90% AUDIENCE: What about like a-- 00:03:15.650 --> 00:03:18.000 align:middle line:84% like in your ear, you have hairs that vibrate 00:03:18.000 --> 00:03:20.180 align:middle line:84% at different frequencies, do you have-- 00:03:20.180 --> 00:03:21.680 align:middle line:84% PROFESSOR: What's the analogy there? 00:03:21.680 --> 00:03:22.260 align:middle line:90% AUDIENCE: Oh. 00:03:22.260 --> 00:03:25.672 align:middle line:84% Like so, if you had a pixels that were listening for light-- 00:03:25.672 --> 00:03:27.880 align:middle line:84% PROFESSOR: Uh-huh, at a particular wavelength, right? 00:03:27.880 --> 00:03:29.830 align:middle line:84% AUDIENCE: Like listening for [INAUDIBLE] light. 00:03:29.830 --> 00:03:32.080 align:middle line:84% PROFESSOR: So imagine you're trying to build a camera. 00:03:32.080 --> 00:03:36.340 align:middle line:84% So right now, I have exactly one pixel, 00:03:36.340 --> 00:03:40.870 align:middle line:84% which gets a signal that comes in at particular hertz, 00:03:40.870 --> 00:03:42.940 align:middle line:84% and then 0 [? at ?] 40 kilohertz, and then 0, 00:03:42.940 --> 00:03:45.680 align:middle line:90% and so on. 00:03:45.680 --> 00:03:49.060 align:middle line:84% So thinking in terms of communication, 00:03:49.060 --> 00:03:53.850 align:middle line:84% we have a carrier, and we have a signal from that. 00:03:53.850 --> 00:03:57.720 align:middle line:84% The 0 [? with ?] 40 kilohertz, [INAUDIBLE] 00:03:57.720 --> 00:04:00.930 align:middle line:84% amplitude modulation and [INAUDIBLE] signal around that. 00:04:00.930 --> 00:04:04.860 align:middle line:84% That's how you are thinking in communication. 00:04:04.860 --> 00:04:07.072 align:middle line:84% And in case of a remote control, you 00:04:07.072 --> 00:04:08.780 align:middle line:84% send 40 kilohertz, nothing, 40 kilohertz. 00:04:08.780 --> 00:04:10.830 align:middle line:90% That's as simple as that-- 00:04:10.830 --> 00:04:12.000 align:middle line:90% a very simple signal. 00:04:12.000 --> 00:04:17.230 align:middle line:84% Now on your-- so you have your [INAUDIBLE].. 00:04:17.230 --> 00:04:20.320 align:middle line:84% If you use a [INAUDIBLE] image [INAUDIBLE] 00:04:20.320 --> 00:04:24.010 align:middle line:84% instead of one pixel, imagine every pixel in a camera 00:04:24.010 --> 00:04:25.580 align:middle line:90% is able to decode the signal. 00:04:25.580 --> 00:04:28.810 align:middle line:84% So instead of taking the 40-kilohertz signal 00:04:28.810 --> 00:04:35.040 align:middle line:84% as a reference carrier, and it all becomes just 1, 0, 1, 0. 00:04:35.040 --> 00:04:36.780 align:middle line:90% That's all [INAUDIBLE]. 00:04:36.780 --> 00:04:41.160 align:middle line:84% Now imagine if I could put a camera so that every pixel has 00:04:41.160 --> 00:04:43.720 align:middle line:90% that property. 00:04:43.720 --> 00:04:45.090 align:middle line:90% So I'm going to put a camera. 00:04:45.090 --> 00:04:47.750 align:middle line:90% 00:04:47.750 --> 00:04:51.830 align:middle line:84% The [INAUDIBLE] pixel here can decode 40 kilohertz 00:04:51.830 --> 00:04:54.440 align:middle line:84% and just pick up what is [? illuminated ?] at 40 00:04:54.440 --> 00:04:57.120 align:middle line:84% kilohertz and ignore what's in the room. 00:04:57.120 --> 00:05:00.710 align:middle line:84% So in a typical room like this, [INAUDIBLE] 00:05:00.710 --> 00:05:02.510 align:middle line:90% you know that this is sunlight. 00:05:02.510 --> 00:05:05.900 align:middle line:84% So sunlight, you know, a huge DC, and then a TV remote 00:05:05.900 --> 00:05:08.300 align:middle line:84% is giving a little bit of signal. 00:05:08.300 --> 00:05:13.220 align:middle line:84% And then all the photodetector does is it just plants it-- 00:05:13.220 --> 00:05:15.480 align:middle line:84% that's just the frequency selection [INAUDIBLE] here. 00:05:15.480 --> 00:05:19.410 align:middle line:84% And this is the signal, and it ignores all the DC. 00:05:19.410 --> 00:05:23.040 align:middle line:84% Now, can I create a camera where every pixel becomes the same 00:05:23.040 --> 00:05:27.840 align:middle line:84% as that, and now I can shine the room with my remote 00:05:27.840 --> 00:05:32.880 align:middle line:84% so that whole scene is being [? flooded ?] by 40 kilohertz, 00:05:32.880 --> 00:05:35.970 align:middle line:84% and in bright daylight this scene will 00:05:35.970 --> 00:05:40.890 align:middle line:84% appear as if it was lit only by this flashing LED 00:05:40.890 --> 00:05:44.180 align:middle line:90% and nothing else? 00:05:44.180 --> 00:05:45.970 align:middle line:90% Is that clear? 00:05:45.970 --> 00:05:47.990 align:middle line:84% AUDIENCE: So there's a lens in here, right? 00:05:47.990 --> 00:05:48.350 align:middle line:90% PROFESSOR: Sorry? 00:05:48.350 --> 00:05:49.670 align:middle line:84% AUDIENCE: There's a lens in this case? 00:05:49.670 --> 00:05:51.545 align:middle line:84% PROFESSOR: Yeah, there's a lens and all that. 00:05:51.545 --> 00:05:53.750 align:middle line:84% It's just a typical camera with a sensor and so on. 00:05:53.750 --> 00:05:55.557 align:middle line:84% This point is being focused here and so on. 00:05:55.557 --> 00:05:56.390 align:middle line:90% It's the same thing. 00:05:56.390 --> 00:05:58.407 align:middle line:84% It shows that the light I'm having here 00:05:58.407 --> 00:06:00.740 align:middle line:84% is [INAUDIBLE] 40 kilohertz and [INAUDIBLE] 40 kilohertz 00:06:00.740 --> 00:06:01.460 align:middle line:90% and so on. 00:06:01.460 --> 00:06:02.918 align:middle line:84% AUDIENCE: But can you build cameras 00:06:02.918 --> 00:06:06.020 align:middle line:84% which operate at 40 hertz, like that [INAUDIBLE] per second? 00:06:06.020 --> 00:06:09.570 align:middle line:84% PROFESSOR: [INAUDIBLE] you could. 00:06:09.570 --> 00:06:13.650 align:middle line:84% So we're not going to get cameras that look like this. 00:06:13.650 --> 00:06:16.055 align:middle line:84% It will happen as the silicon improves and so on. 00:06:16.055 --> 00:06:18.680 align:middle line:84% Of course, there's always people that [INAUDIBLE] 40 kilohertz. 00:06:18.680 --> 00:06:22.610 align:middle line:84% Now imagine, somebody gives me a flashlight that actually 00:06:22.610 --> 00:06:24.172 align:middle line:90% runs at [? 2 ?] kilohertz. 00:06:24.172 --> 00:06:26.830 align:middle line:90% 00:06:26.830 --> 00:06:30.990 align:middle line:84% And this one runs at 40 kilohertz. 00:06:30.990 --> 00:06:32.820 align:middle line:84% And this particular pixel actually 00:06:32.820 --> 00:06:37.770 align:middle line:84% captures the signal across 14 and 15, 00:06:37.770 --> 00:06:41.100 align:middle line:84% and in software, it can decide what's the [INAUDIBLE] at 14 00:06:41.100 --> 00:06:43.355 align:middle line:84% and what's the [INAUDIBLE] at 15. 00:06:43.355 --> 00:06:44.730 align:middle line:84% What did you just [? look ?] here 00:06:44.730 --> 00:06:47.500 align:middle line:90% compared to assignment number 1? 00:06:47.500 --> 00:06:51.150 align:middle line:84% Two flashlights on at the same time. 00:06:51.150 --> 00:06:54.360 align:middle line:84% I want to know how the signals [INAUDIBLE].. 00:06:54.360 --> 00:06:56.788 align:middle line:90% This one and this one. 00:06:56.788 --> 00:07:00.750 align:middle line:84% So this is A and this is B. And the image I'm getting 00:07:00.750 --> 00:07:02.370 align:middle line:90% is A plus B. 00:07:02.370 --> 00:07:06.240 align:middle line:84% But in software, I can decompose and say which part 00:07:06.240 --> 00:07:07.420 align:middle line:90% of the imaging-- 00:07:07.420 --> 00:07:10.290 align:middle line:84% which intensity came because of A and which intensity 00:07:10.290 --> 00:07:13.900 align:middle line:84% came because of B. So software, I 00:07:13.900 --> 00:07:16.990 align:middle line:84% can tune between this light source and that light source, 00:07:16.990 --> 00:07:18.400 align:middle line:90% just like I can-- 00:07:18.400 --> 00:07:22.420 align:middle line:84% on your car radio, you can tune between 99 megahertz station 00:07:22.420 --> 00:07:24.770 align:middle line:90% and the 80 megahertz station. 00:07:24.770 --> 00:07:29.540 align:middle line:84% So we will tune that on the camera. 00:07:29.540 --> 00:07:31.960 align:middle line:84% And once we have that, imagine cinematography. 00:07:31.960 --> 00:07:34.720 align:middle line:84% You can put all kinds of lights through the movie, 00:07:34.720 --> 00:07:36.970 align:middle line:84% and then go in Photoshop and change any light, 00:07:36.970 --> 00:07:39.880 align:middle line:90% any color, any intensity. 00:07:39.880 --> 00:07:42.166 align:middle line:84% Again, beautiful light [INAUDIBLE].. 00:07:42.166 --> 00:07:45.270 align:middle line:84% AUDIENCE: But again, lots of data [INAUDIBLE].. 00:07:45.270 --> 00:07:47.676 align:middle line:84% PROFESSOR: Yeah, but it doesn't [INAUDIBLE].. 00:07:47.676 --> 00:07:48.662 align:middle line:90% [LAUGHTER] 00:07:48.662 --> 00:07:50.110 align:middle line:90% [INTERPOSING VOICES] 00:07:50.110 --> 00:07:53.810 align:middle line:90% 00:07:53.810 --> 00:07:56.150 align:middle line:84% And they'll be happy to clear [INAUDIBLE].. 00:07:56.150 --> 00:07:59.460 align:middle line:90% [LAUGHTER] 00:07:59.460 --> 00:08:00.710 align:middle line:90% So there's a lot more to come. 00:08:00.710 --> 00:08:03.670 align:middle line:84% So every time you think about how light interacts 00:08:03.670 --> 00:08:09.580 align:middle line:84% with the world, say, how can I use that for imaging? 00:08:09.580 --> 00:08:11.320 align:middle line:90% AUDIENCE: Is it crazy-- 00:08:11.320 --> 00:08:14.080 align:middle line:84% is that kind of how sonar works, or are they just 00:08:14.080 --> 00:08:15.280 align:middle line:90% like [INAUDIBLE]? 00:08:15.280 --> 00:08:19.850 align:middle line:84% PROFESSOR: What's weird is I can take sound and create images 00:08:19.850 --> 00:08:20.350 align:middle line:90% like-- 00:08:20.350 --> 00:08:21.160 align:middle line:90% AUDIENCE: You meant sonar. 00:08:21.160 --> 00:08:21.550 align:middle line:90% PROFESSOR: Sonar. 00:08:21.550 --> 00:08:22.480 align:middle line:90% AUDIENCE: Boop. 00:08:22.480 --> 00:08:23.327 align:middle line:90% Yeah, yeah, yeah. 00:08:23.327 --> 00:08:24.160 align:middle line:90% [INTERPOSING VOICES] 00:08:24.160 --> 00:08:26.020 align:middle line:90% AUDIENCE: Or LiDAR. 00:08:26.020 --> 00:08:29.230 align:middle line:84% PROFESSOR: LiDAR-- yeah, all those methods 00:08:29.230 --> 00:08:31.885 align:middle line:84% are basically using the principles for [INAUDIBLE].. 00:08:31.885 --> 00:08:35.530 align:middle line:84% It starts, it bounces, it has certain properties 00:08:35.530 --> 00:08:37.450 align:middle line:84% in terms of presence or [? absence, ?] 00:08:37.450 --> 00:08:41.620 align:middle line:84% position, color, space modulation, time modulation, 00:08:41.620 --> 00:08:42.502 align:middle line:90% and all these things. 00:08:42.502 --> 00:08:43.960 align:middle line:84% AUDIENCE: Just a little [? silly ?] 00:08:43.960 --> 00:08:45.880 align:middle line:84% kind of question-- who was the first person 00:08:45.880 --> 00:08:48.273 align:middle line:90% to do computational photography? 00:08:48.273 --> 00:08:50.190 align:middle line:84% PROFESSOR: Steve Mann right here [INAUDIBLE].. 00:08:50.190 --> 00:08:52.500 align:middle line:84% Steve Mann and Ros Picard were the first one 00:08:52.500 --> 00:08:54.640 align:middle line:84% to use the term "computational photography," 00:08:54.640 --> 00:08:57.360 align:middle line:84% although they used it in a very specific context 00:08:57.360 --> 00:08:59.700 align:middle line:84% for high dynamic [? range ?] imaging. 00:08:59.700 --> 00:09:03.660 align:middle line:84% And then later on, very limited-- 00:09:03.660 --> 00:09:06.690 align:middle line:84% very important people, pioneers in the field 00:09:06.690 --> 00:09:11.235 align:middle line:84% such as Shree Nayar, and Marc Levoy, and so on. 00:09:11.235 --> 00:09:15.210 align:middle line:84% They were [INAUDIBLE] even before the term was around. 00:09:15.210 --> 00:09:18.600 align:middle line:84% Actually, when you look at all these papers and presentations, 00:09:18.600 --> 00:09:20.880 align:middle line:84% I would say over half of them are just 00:09:20.880 --> 00:09:22.260 align:middle line:90% because of those two guys. 00:09:22.260 --> 00:09:24.052 align:middle line:84% AUDIENCE: Because when you talk about this, 00:09:24.052 --> 00:09:26.700 align:middle line:84% it's also when NASA explores planets, 00:09:26.700 --> 00:09:31.528 align:middle line:84% you also think of [INAUDIBLE],, what does it mean, [INAUDIBLE].. 00:09:31.528 --> 00:09:32.320 align:middle line:90% PROFESSOR: Exactly. 00:09:32.320 --> 00:09:36.270 align:middle line:84% I mean, what we're talking about is really 00:09:36.270 --> 00:09:37.950 align:middle line:90% communication concepts. 00:09:37.950 --> 00:09:39.750 align:middle line:90% It's 100 years old. 00:09:39.750 --> 00:09:42.450 align:middle line:84% So a lot of concepts kind of get borrowed. 00:09:42.450 --> 00:09:45.810 align:middle line:84% And [INAUDIBLE],, you couldn't think 00:09:45.810 --> 00:09:49.680 align:middle line:84% of decoding a particular hertz signal, right? 00:09:49.680 --> 00:09:54.220 align:middle line:84% And we have moved to digital only what, 15, 20 years ago? 00:09:54.220 --> 00:09:57.990 align:middle line:84% And astronomy, all that math and all the techniques 00:09:57.990 --> 00:10:03.090 align:middle line:84% that we use in communication become possible in our world. 00:10:03.090 --> 00:10:05.160 align:middle line:90% So this is kind of a-- 00:10:05.160 --> 00:10:06.630 align:middle line:84% because at the same time, when you 00:10:06.630 --> 00:10:10.260 align:middle line:84% are in the communication world, the signals 00:10:10.260 --> 00:10:12.090 align:middle line:90% don't have very high dimension. 00:10:12.090 --> 00:10:13.860 align:middle line:84% It's usually a two-dimensional signal, 00:10:13.860 --> 00:10:18.840 align:middle line:84% and a number of stations, and a frequency range-- 00:10:18.840 --> 00:10:20.370 align:middle line:84% basically a two-dimensional signal. 00:10:20.370 --> 00:10:22.200 align:middle line:84% Every trans-- every radio station 00:10:22.200 --> 00:10:23.910 align:middle line:90% is transmitting from audio. 00:10:23.910 --> 00:10:25.470 align:middle line:90% And audio is one dimensional. 00:10:25.470 --> 00:10:30.010 align:middle line:84% And since it's a two-dimensional signal that's in our world, 00:10:30.010 --> 00:10:32.220 align:middle line:84% and we capture that on our antenna 00:10:32.220 --> 00:10:34.170 align:middle line:84% as a one-dimensional signal over time, 00:10:34.170 --> 00:10:37.380 align:middle line:84% and we decode that and record back the [INAUDIBLE].. 00:10:37.380 --> 00:10:39.660 align:middle line:84% So usually it's not very high dimensional. 00:10:39.660 --> 00:10:42.430 align:middle line:84% And even if it had high dimension, they're multi-scale. 00:10:42.430 --> 00:10:45.630 align:middle line:84% So just if I'm sending 500 channels on a fiber, 00:10:45.630 --> 00:10:47.430 align:middle line:84% that's just 500 separate signals. 00:10:47.430 --> 00:10:51.520 align:middle line:84% They are not intermixed like we have here. 00:10:51.520 --> 00:10:55.680 align:middle line:84% So although [INAUDIBLE] similar, the problems in imaging 00:10:55.680 --> 00:10:58.580 align:middle line:90% are more complex. 00:10:58.580 --> 00:11:03.250 align:middle line:84% The high dimensional have [INAUDIBLE] problems 00:11:03.250 --> 00:11:05.800 align:middle line:90% [INAUDIBLE]. 00:11:05.800 --> 00:11:09.260 align:middle line:84% But research is all about fusion of the similar. 00:11:09.260 --> 00:11:12.730 align:middle line:84% So if you can learn ideas from communication, and optics, 00:11:12.730 --> 00:11:15.370 align:middle line:84% and quantum computing, and signal processing, 00:11:15.370 --> 00:11:18.670 align:middle line:84% you put all that together and mix, and you all this-- you 00:11:18.670 --> 00:11:19.510 align:middle line:90% can create magic. 00:11:19.510 --> 00:11:25.330 align:middle line:84% And almost every project we see has some element of magic. 00:11:25.330 --> 00:11:28.700 align:middle line:84% And that's what makes it very exciting. 00:11:28.700 --> 00:11:31.460 align:middle line:84% AUDIENCE: So 40 kilohertz seems quite fast. 00:11:31.460 --> 00:11:33.320 align:middle line:84% But it seems like maybe we don't actually 00:11:33.320 --> 00:11:34.993 align:middle line:90% need to be quite that fast. 00:11:34.993 --> 00:11:35.910 align:middle line:90% PROFESSOR: Yeah, yeah. 00:11:35.910 --> 00:11:36.535 align:middle line:90% AUDIENCE: Yeah. 00:11:36.535 --> 00:11:40.010 align:middle line:84% So I'm just trying to think, what would the-- 00:11:40.010 --> 00:11:42.950 align:middle line:84% how slow could you go and still, basically, 00:11:42.950 --> 00:11:45.478 align:middle line:90% eliminate the DC component? 00:11:45.478 --> 00:11:47.270 align:middle line:84% I mean, it's part of the fluorescent lights 00:11:47.270 --> 00:11:49.873 align:middle line:90% that are 60 hertz or-- 00:11:49.873 --> 00:11:50.540 align:middle line:90% PROFESSOR: Yeah. 00:11:50.540 --> 00:11:53.558 align:middle line:84% I mean, [INAUDIBLE] like, what, 25 kilohertz 00:11:53.558 --> 00:11:57.050 align:middle line:90% to remove the flicker? 00:11:57.050 --> 00:11:58.820 align:middle line:90% But yeah, you could use-- 00:11:58.820 --> 00:12:01.130 align:middle line:84% I mean, if the camera's 60 hertz, 00:12:01.130 --> 00:12:05.420 align:middle line:84% you could just use a 60-hertz strobe and turn it on in one 00:12:05.420 --> 00:12:08.420 align:middle line:84% frame and off-- on every odd frame and even-- 00:12:08.420 --> 00:12:09.660 align:middle line:90% off in every even frame. 00:12:09.660 --> 00:12:12.990 align:middle line:84% And that alone will allow you to do this subtraction. 00:12:12.990 --> 00:12:16.460 align:middle line:84% So the only problem is that if you do a pure subtraction, 00:12:16.460 --> 00:12:19.430 align:middle line:84% you're going to subtract really two large quantities, two 00:12:19.430 --> 00:12:20.400 align:middle line:90% large numbers. 00:12:20.400 --> 00:12:25.100 align:middle line:84% So in the first image, you have sun [INAUDIBLE] [? mesh. ?] 00:12:25.100 --> 00:12:27.273 align:middle line:84% And the second image is just the sun. 00:12:27.273 --> 00:12:29.190 align:middle line:84% And this is very, very small compared to this. 00:12:29.190 --> 00:12:31.340 align:middle line:84% You're subtracting two large numbers 00:12:31.340 --> 00:12:34.190 align:middle line:84% and expecting to recover the contribution because 00:12:34.190 --> 00:12:38.225 align:middle line:84% of the [? mesh. ?] Is that [INAUDIBLE]?? 00:12:38.225 --> 00:12:41.853 align:middle line:84% AUDIENCE: Sounds like an error accumulation problem. 00:12:41.853 --> 00:12:42.520 align:middle line:90% PROFESSOR: Yeah. 00:12:42.520 --> 00:12:45.180 align:middle line:84% But that's [? exactly ?] the problem, 00:12:45.180 --> 00:12:47.450 align:middle line:90% communication [INAUDIBLE]. 00:12:47.450 --> 00:12:50.210 align:middle line:84% The carriers and the signal is so tiny 00:12:50.210 --> 00:12:54.620 align:middle line:84% that's riding in free space over large distances that they use 00:12:54.620 --> 00:12:58.456 align:middle line:84% really clever coding mechanisms so that your imaging 00:12:58.456 --> 00:13:00.370 align:middle line:84% [INAUDIBLE] increase [INAUDIBLE].. 00:13:00.370 --> 00:13:03.195 align:middle line:90% 00:13:03.195 --> 00:13:04.820 align:middle line:84% So I just want you to think very broad. 00:13:04.820 --> 00:13:08.420 align:middle line:84% I know many of you here have very interesting backgrounds 00:13:08.420 --> 00:13:12.290 align:middle line:84% in communication, and chemistry, and interaction, and so on. 00:13:12.290 --> 00:13:14.600 align:middle line:90% So try to make the best of that. 00:13:14.600 --> 00:13:17.400 align:middle line:90% 00:13:17.400 --> 00:13:19.130 align:middle line:84% So temporal modulation, actually, 00:13:19.130 --> 00:13:23.750 align:middle line:84% is not used that effectively right now in imaging. 00:13:23.750 --> 00:13:26.730 align:middle line:84% So [? certain ?] projects, I'm not going to go into detail, 00:13:26.730 --> 00:13:30.380 align:middle line:84% but they're on that wiki that I sent you. 00:13:30.380 --> 00:13:32.330 align:middle line:84% So please add more information there. 00:13:32.330 --> 00:13:34.520 align:middle line:84% Add your own experiences, some of the things 00:13:34.520 --> 00:13:37.340 align:middle line:84% you are mentioning, some of the projects you're mentioning. 00:13:37.340 --> 00:13:40.248 align:middle line:84% Please go and add all those things to those wikis. 00:13:40.248 --> 00:13:47.670 align:middle line:90% 00:13:47.670 --> 00:13:48.180 align:middle line:90% All right. 00:13:48.180 --> 00:13:50.580 align:middle line:84% So sometimes, you can't control the illumination, 00:13:50.580 --> 00:13:54.700 align:middle line:84% but you can just exploit natural illumination. 00:13:54.700 --> 00:13:55.270 align:middle line:90% OK. 00:13:55.270 --> 00:13:59.110 align:middle line:84% So here's a project from Washington University, 00:13:59.110 --> 00:14:00.850 align:middle line:90% St. Louis. 00:14:00.850 --> 00:14:05.080 align:middle line:84% And what they did was they took webcam images 00:14:05.080 --> 00:14:09.100 align:middle line:84% all day long at a given time of the day. 00:14:09.100 --> 00:14:13.550 align:middle line:84% So on the x-axis, you have time of day. 00:14:13.550 --> 00:14:18.520 align:middle line:84% So it's dark in the night, then daytime, and again dark. 00:14:18.520 --> 00:14:22.280 align:middle line:84% And on the y-axis, you have day of the year. 00:14:22.280 --> 00:14:27.250 align:middle line:84% So those are how many-- yeah, I guess just day of the year. 00:14:27.250 --> 00:14:31.810 align:middle line:84% I don't know after many days each was calculated. 00:14:31.810 --> 00:14:36.430 align:middle line:84% If the top is 1st of January and bottom is 31st of December, 00:14:36.430 --> 00:14:40.300 align:middle line:84% what can you say from this data set? 00:14:40.300 --> 00:14:42.040 align:middle line:84% AUDIENCE: Winter has shorter days. 00:14:42.040 --> 00:14:43.840 align:middle line:84% PROFESSOR: Winter has shorter days, which 00:14:43.840 --> 00:14:45.038 align:middle line:90% means where is this camera? 00:14:45.038 --> 00:14:46.580 align:middle line:84% AUDIENCE: In the northern hemisphere. 00:14:46.580 --> 00:14:49.340 align:middle line:84% PROFESSOR: It's in the northern hemisphere, right? 00:14:49.340 --> 00:14:51.560 align:middle line:84% And you can probably say more about 00:14:51.560 --> 00:14:54.390 align:middle line:84% if we just had the ratio of the smallest day to the largest-- 00:14:54.390 --> 00:14:57.830 align:middle line:84% longest day, that will tell you the latitude, 00:14:57.830 --> 00:14:59.420 align:middle line:84% because when you're on the equator, 00:14:59.420 --> 00:15:03.470 align:middle line:84% the longest and shortest days have equal length. 00:15:03.470 --> 00:15:06.160 align:middle line:90% But as you go away from that-- 00:15:06.160 --> 00:15:08.600 align:middle line:84% there are already a lot of data embedded 00:15:08.600 --> 00:15:10.745 align:middle line:90% in this natural illumination. 00:15:10.745 --> 00:15:12.600 align:middle line:84% So this project is really beautiful. 00:15:12.600 --> 00:15:14.400 align:middle line:84% They did all kinds of interesting things. 00:15:14.400 --> 00:15:16.550 align:middle line:84% So they have hundreds of static-- thousands 00:15:16.550 --> 00:15:21.650 align:middle line:84% of static cameras, variation over the year, over a day. 00:15:21.650 --> 00:15:22.850 align:middle line:90% They put all that together. 00:15:22.850 --> 00:15:26.030 align:middle line:90% 00:15:26.030 --> 00:15:28.350 align:middle line:84% They can do really interesting things. 00:15:28.350 --> 00:15:35.476 align:middle line:84% So it turns out in a traditional lighting, 00:15:35.476 --> 00:15:39.210 align:middle line:84% in a typical scenario, light is linear. 00:15:39.210 --> 00:15:41.380 align:middle line:90% What does it mean? 00:15:41.380 --> 00:15:44.400 align:middle line:84% It means that if I have a scene, I light it 00:15:44.400 --> 00:15:48.660 align:middle line:84% with particular brightness, a particular intensity of light, 00:15:48.660 --> 00:15:49.890 align:middle line:90% I get certain brightness. 00:15:49.890 --> 00:15:53.190 align:middle line:84% If I make my light twice as bright, 00:15:53.190 --> 00:15:57.180 align:middle line:84% everything will become twice as bright. 00:15:57.180 --> 00:15:58.740 align:middle line:90% As simple as that. 00:15:58.740 --> 00:16:02.010 align:middle line:84% This is not true at all the intensities of light. 00:16:02.010 --> 00:16:04.710 align:middle line:84% When you go really, really bright light, it's not true. 00:16:04.710 --> 00:16:08.520 align:middle line:84% The world starts behaving in a nonlinear fashion. 00:16:08.520 --> 00:16:14.130 align:middle line:84% If you have your speaker on your synthesizer, 00:16:14.130 --> 00:16:20.670 align:middle line:84% if you pump twice the power to your speaker, 00:16:20.670 --> 00:16:22.635 align:middle line:84% do you always get twice the loudness? 00:16:22.635 --> 00:16:25.470 align:middle line:90% 00:16:25.470 --> 00:16:28.470 align:middle line:84% AUDIENCE: It only increases 1 decibel. 00:16:28.470 --> 00:16:29.927 align:middle line:90% AUDIENCE: It begins to saturate. 00:16:29.927 --> 00:16:31.260 align:middle line:90% PROFESSOR: It tends to saturate. 00:16:31.260 --> 00:16:35.785 align:middle line:84% And eventually, you'll run into nonlinear behavior. 00:16:35.785 --> 00:16:37.660 align:middle line:84% And the same thing is true for light as well. 00:16:37.660 --> 00:16:40.320 align:middle line:84% But as far as sunlight is concerned and the type of world 00:16:40.320 --> 00:16:42.240 align:middle line:84% we are involved, everything is linear, 00:16:42.240 --> 00:16:43.870 align:middle line:84% so we don't have to worry about it. 00:16:43.870 --> 00:16:45.300 align:middle line:84% And because everything is linear, 00:16:45.300 --> 00:16:47.820 align:middle line:84% mathematically it can all be expressed 00:16:47.820 --> 00:16:51.390 align:middle line:84% as just linear transforms, and linear algebra, and so on. 00:16:51.390 --> 00:16:53.670 align:middle line:84% That's why background in linear algebra 00:16:53.670 --> 00:16:57.300 align:middle line:84% is very useful when you're doing any imaging work. 00:16:57.300 --> 00:16:58.800 align:middle line:84% So they did some very simple things. 00:16:58.800 --> 00:17:01.590 align:middle line:84% Like they took all these images, just 00:17:01.590 --> 00:17:05.910 align:middle line:84% did a PCA, Principal Component Analysis. 00:17:05.910 --> 00:17:09.180 align:middle line:84% And that image allows them to figure out 00:17:09.180 --> 00:17:14.020 align:middle line:84% the haze, and cloud, and the orientation of the surfaces. 00:17:14.020 --> 00:17:15.300 align:middle line:90% So this essentially lists-- 00:17:15.300 --> 00:17:17.970 align:middle line:84% and I believe they can figure out that this building is 00:17:17.970 --> 00:17:20.680 align:middle line:84% facing one way versus this building, 00:17:20.680 --> 00:17:24.000 align:middle line:84% and so on, just without even analyzing and doing 00:17:24.000 --> 00:17:25.740 align:middle line:84% any sophisticated computer vision, 00:17:25.740 --> 00:17:27.960 align:middle line:84% just from the sequence of images. 00:17:27.960 --> 00:17:29.820 align:middle line:84% And then they can segment the scene. 00:17:29.820 --> 00:17:32.010 align:middle line:90% This is something close. 00:17:32.010 --> 00:17:35.970 align:middle line:84% And with distance very far away, they can encode that. 00:17:35.970 --> 00:17:40.830 align:middle line:84% And they can even figure out where a webcam is, its latitude 00:17:40.830 --> 00:17:43.320 align:middle line:90% and longitude. 00:17:43.320 --> 00:17:48.240 align:middle line:84% And Robert [INAUDIBLE] told me that they can do-- 00:17:48.240 --> 00:17:49.920 align:middle line:84% just based on the sunrise and sunset 00:17:49.920 --> 00:17:52.110 align:middle line:84% data set that we saw earlier, they 00:17:52.110 --> 00:17:54.605 align:middle line:84% can localize with 50-mile accuracy. 00:17:54.605 --> 00:17:57.515 align:middle line:90% 00:17:57.515 --> 00:18:00.300 align:middle line:84% And if you have some speed cameras where 00:18:00.300 --> 00:18:03.370 align:middle line:84% you know the locations, then you can interpolate and go down 00:18:03.370 --> 00:18:05.410 align:middle line:90% to about 25 miles. 00:18:05.410 --> 00:18:07.690 align:middle line:84% And in addition, if you have satellite imagery, 00:18:07.690 --> 00:18:09.940 align:middle line:84% so you know how the intensity is changing, 00:18:09.940 --> 00:18:12.010 align:middle line:90% then you can do around 15 miles. 00:18:12.010 --> 00:18:16.030 align:middle line:84% And then the people at CMU such as Srinivasa Narasimhan 00:18:16.030 --> 00:18:20.110 align:middle line:84% and Alyosha Efros, they recently did a paper where they can just 00:18:20.110 --> 00:18:23.210 align:middle line:90% look at a patch in the sky. 00:18:23.210 --> 00:18:27.260 align:middle line:84% And if you look at a patch in the sky in broad daylight, 00:18:27.260 --> 00:18:29.410 align:middle line:90% it always has a gradient. 00:18:29.410 --> 00:18:31.600 align:middle line:84% And depending on where the sun is, 00:18:31.600 --> 00:18:33.610 align:middle line:84% the gradient has a particular orientation 00:18:33.610 --> 00:18:36.310 align:middle line:90% in x or y, the intensity ramp. 00:18:36.310 --> 00:18:39.910 align:middle line:84% And that actually localizes the direction of the sun. 00:18:39.910 --> 00:18:42.640 align:middle line:84% So now they can look at webcam images 00:18:42.640 --> 00:18:47.050 align:middle line:84% and click on the part that shows the sky, 00:18:47.050 --> 00:18:48.970 align:middle line:84% and they can localize the cameras down 00:18:48.970 --> 00:18:51.915 align:middle line:90% to, again, a few tens of miles. 00:18:51.915 --> 00:18:53.290 align:middle line:84% I forget exactly what the numbers 00:18:53.290 --> 00:18:58.110 align:middle line:90% are, but pretty fascinating. 00:18:58.110 --> 00:19:00.270 align:middle line:84% And they're not even using polarization. 00:19:00.270 --> 00:19:03.900 align:middle line:84% If you use polarization, it can get even better, 00:19:03.900 --> 00:19:06.600 align:middle line:84% because the sky is highly polarized. 00:19:06.600 --> 00:19:08.820 align:middle line:84% AUDIENCE: I have a question about it, actually. 00:19:08.820 --> 00:19:13.200 align:middle line:84% Do normal digital camera sensors or film or anything, 00:19:13.200 --> 00:19:17.107 align:middle line:84% do they have any polarization dependence at all? 00:19:17.107 --> 00:19:18.690 align:middle line:84% PROFESSOR: An ordinary sensor doesn't. 00:19:18.690 --> 00:19:19.980 align:middle line:84% But you can always put a polarization-- 00:19:19.980 --> 00:19:21.340 align:middle line:84% AUDIENCE: You can always put a polarizing filter, 00:19:21.340 --> 00:19:23.820 align:middle line:84% but there's really no correlation [INAUDIBLE]?? 00:19:23.820 --> 00:19:25.120 align:middle line:90% PROFESSOR: As far as I know. 00:19:25.120 --> 00:19:26.040 align:middle line:90% Yeah. 00:19:26.040 --> 00:19:32.250 align:middle line:84% Even the human eye does not have very strong sensitivity 00:19:32.250 --> 00:19:33.360 align:middle line:90% to the polarization. 00:19:33.360 --> 00:19:35.820 align:middle line:90% But there are some results-- 00:19:35.820 --> 00:19:38.640 align:middle line:84% and if you talk to Matt Hirsch, he claims he knows-- 00:19:38.640 --> 00:19:40.710 align:middle line:90% he can see polarization. 00:19:40.710 --> 00:19:42.793 align:middle line:84% He even has experiments where if you see one way, 00:19:42.793 --> 00:19:44.460 align:middle line:84% you see one color, and you the other way 00:19:44.460 --> 00:19:45.600 align:middle line:90% you see a different color. 00:19:45.600 --> 00:19:47.100 align:middle line:84% He's shown it to me dozens of times, 00:19:47.100 --> 00:19:49.125 align:middle line:90% but I don't see the difference. 00:19:49.125 --> 00:19:51.000 align:middle line:84% But he's been able to recruit a lot of people 00:19:51.000 --> 00:19:54.480 align:middle line:90% to say yes, they see it. 00:19:54.480 --> 00:19:57.300 align:middle line:84% And there are very few actually animal eyes 00:19:57.300 --> 00:19:59.112 align:middle line:90% can sense polarization. 00:19:59.112 --> 00:20:00.570 align:middle line:84% There are some underwater creatures 00:20:00.570 --> 00:20:02.610 align:middle line:84% that can do a pretty good job, for instance. 00:20:02.610 --> 00:20:08.003 align:middle line:90% 00:20:08.003 --> 00:20:09.420 align:middle line:84% So again, they can do the encoding 00:20:09.420 --> 00:20:16.920 align:middle line:84% of that, how far other things are, orientation of surfaces. 00:20:16.920 --> 00:20:19.320 align:middle line:84% So here, you can see that orientation-- this 00:20:19.320 --> 00:20:20.570 align:middle line:90% is different from orientation. 00:20:20.570 --> 00:20:23.690 align:middle line:84% That-- how would you figure that out, by the way? 00:20:23.690 --> 00:20:24.740 align:middle line:90% AUDIENCE: Shadows. 00:20:24.740 --> 00:20:27.500 align:middle line:84% PROFESSOR: Shadows and sunlight, because some faces 00:20:27.500 --> 00:20:30.410 align:middle line:84% will be lit [INAUDIBLE] than others 00:20:30.410 --> 00:20:32.250 align:middle line:90% depending on time of day. 00:20:32.250 --> 00:20:35.480 align:middle line:84% So you don't have to process it in an individual manner. 00:20:35.480 --> 00:20:38.390 align:middle line:84% You just throw it in a big matrix [INAUDIBLE] PCA, 00:20:38.390 --> 00:20:39.839 align:middle line:90% and [INAUDIBLE]. 00:20:39.839 --> 00:20:42.720 align:middle line:90% 00:20:42.720 --> 00:20:43.220 align:middle line:90% OK. 00:20:43.220 --> 00:20:45.930 align:middle line:84% So let me-- we saw this example last time. 00:20:45.930 --> 00:20:49.300 align:middle line:90% So I'll skip that. 00:20:49.300 --> 00:20:53.021 align:middle line:84% Let me switch to light fields and talk about our assignment. 00:20:53.021 --> 00:20:58.320 align:middle line:90% 00:20:58.320 --> 00:20:58.970 align:middle line:90% All right. 00:20:58.970 --> 00:21:02.690 align:middle line:90% So light fields. 00:21:02.690 --> 00:21:04.880 align:middle line:84% It's one of the most important concepts we're 00:21:04.880 --> 00:21:07.112 align:middle line:90% going to learn in this class. 00:21:07.112 --> 00:21:09.320 align:middle line:84% And again, realizing that the appearance of the world 00:21:09.320 --> 00:21:12.860 align:middle line:84% is higher dimensional, not two dimensional. 00:21:12.860 --> 00:21:14.090 align:middle line:90% You have a 3D world. 00:21:14.090 --> 00:21:18.500 align:middle line:84% You project it on a 2D image, clearly a lot of information 00:21:18.500 --> 00:21:21.380 align:middle line:90% is lost. 00:21:21.380 --> 00:21:26.810 align:middle line:84% Now, if you build a so-called plenoptic function, which is-- 00:21:26.810 --> 00:21:30.860 align:middle line:84% what is set of all things we can ever see? 00:21:30.860 --> 00:21:34.070 align:middle line:84% It was a name, actually, given by Ted Adelson, a professor 00:21:34.070 --> 00:21:36.710 align:middle line:90% here in the early '90s. 00:21:36.710 --> 00:21:40.310 align:middle line:84% Then it turns out it's a very high-dimensional world. 00:21:40.310 --> 00:21:43.280 align:middle line:84% If I stay in one place and think about the bubble around me, 00:21:43.280 --> 00:21:48.710 align:middle line:84% I have the azimuth and elevation of every direction, just 00:21:48.710 --> 00:21:49.795 align:middle line:90% the bubble-- 00:21:49.795 --> 00:21:52.790 align:middle line:84% on Google Street Map, you have a bubble for every location. 00:21:52.790 --> 00:21:57.140 align:middle line:84% That's [? three ?] times [? too. ?] 00:21:57.140 --> 00:22:02.420 align:middle line:84% And that's over time and over wavelength, different colors 00:22:02.420 --> 00:22:03.340 align:middle line:90% and over time. 00:22:03.340 --> 00:22:04.660 align:middle line:90% So that's four dimensional. 00:22:04.660 --> 00:22:07.570 align:middle line:84% Now, I can put these bubbles in different places. 00:22:07.570 --> 00:22:10.360 align:middle line:84% And every bubble can be placed in x, y, z. 00:22:10.360 --> 00:22:13.670 align:middle line:84% So there are three additional degrees of freedom. 00:22:13.670 --> 00:22:17.960 align:middle line:84% And if you can capture all that information, 00:22:17.960 --> 00:22:20.980 align:middle line:84% then you can recreate a movie from any viewpoint 00:22:20.980 --> 00:22:24.640 align:middle line:90% at any time at any wavelength. 00:22:24.640 --> 00:22:26.140 align:middle line:84% But it's extremely high dimensional. 00:22:26.140 --> 00:22:29.890 align:middle line:90% This is seven dimensional. 00:22:29.890 --> 00:22:33.860 align:middle line:84% So the world is actually seven dimensional. 00:22:33.860 --> 00:22:40.840 align:middle line:84% And if somebody built this magical device, 00:22:40.840 --> 00:22:43.510 align:middle line:84% it will have-- if we make a [INAUDIBLE].. 00:22:43.510 --> 00:22:45.400 align:middle line:84% Now we're going to simplify that. 00:22:45.400 --> 00:22:49.180 align:middle line:84% And let's say, OK, for all these bubbles shown in the blue, 00:22:49.180 --> 00:22:54.040 align:middle line:84% all the rays are emanating, and if I think of any point 00:22:54.040 --> 00:22:56.500 align:middle line:84% in the world-- and for now, we're going to ignore time 00:22:56.500 --> 00:22:58.520 align:middle line:90% and wavelength-- 00:22:58.520 --> 00:23:00.520 align:middle line:84% it becomes five dimensional, from seven to five, 00:23:00.520 --> 00:23:02.410 align:middle line:84% because we ignore time and wavelength. 00:23:02.410 --> 00:23:04.210 align:middle line:90% I can take a point in 3D. 00:23:04.210 --> 00:23:09.310 align:middle line:84% And from that point in 3D, I can think of a direction. 00:23:09.310 --> 00:23:11.020 align:middle line:84% And the direction is only 2D, not 3D. 00:23:11.020 --> 00:23:11.620 align:middle line:90% Why is that? 00:23:11.620 --> 00:23:14.320 align:middle line:90% 00:23:14.320 --> 00:23:16.375 align:middle line:90% x, y, z for position-- 00:23:16.375 --> 00:23:17.500 align:middle line:90% AUDIENCE: Theta [INAUDIBLE] 00:23:17.500 --> 00:23:19.600 align:middle line:84% PROFESSOR: But only theta phi for angle. 00:23:19.600 --> 00:23:21.092 align:middle line:90% Why is it not three dimensional? 00:23:21.092 --> 00:23:24.552 align:middle line:90% 00:23:24.552 --> 00:23:26.760 align:middle line:84% AUDIENCE: What would you use the third dimension for? 00:23:26.760 --> 00:23:28.135 align:middle line:84% PROFESSOR: Because the roll along 00:23:28.135 --> 00:23:30.510 align:middle line:90% the ray does not really matter. 00:23:30.510 --> 00:23:32.202 align:middle line:84% So you have yaw, pitch, and roll, 00:23:32.202 --> 00:23:34.410 align:middle line:84% but the roll can be ignored, because the intensity is 00:23:34.410 --> 00:23:35.845 align:middle line:90% the same even if you have roll. 00:23:35.845 --> 00:23:39.030 align:middle line:90% So it's only five dimensional. 00:23:39.030 --> 00:23:44.760 align:middle line:84% But then if you have an occluder here, then 00:23:44.760 --> 00:23:47.430 align:middle line:84% the intensity of this ray is different from intensity 00:23:47.430 --> 00:23:48.826 align:middle line:90% of this ray. 00:23:48.826 --> 00:23:52.920 align:middle line:84% If you have no occluder, then the ray intensity 00:23:52.920 --> 00:23:54.550 align:middle line:90% remains the same. 00:23:54.550 --> 00:23:58.530 align:middle line:84% So now, actually you can go down to just four images rather than 00:23:58.530 --> 00:24:01.260 align:middle line:90% five. 00:24:01.260 --> 00:24:08.790 align:middle line:84% So the space of all lines in 3D is actually four dimensional 00:24:08.790 --> 00:24:11.650 align:middle line:84% if you want to express all the rays, 00:24:11.650 --> 00:24:14.174 align:middle line:84% then it's just four dimensional-- 00:24:14.174 --> 00:24:16.420 align:middle line:84% ax plus b y plus cz plus [? p. ?] 00:24:16.420 --> 00:24:18.240 align:middle line:90% There's four [INAUDIBLE]. 00:24:18.240 --> 00:24:20.940 align:middle line:90% 00:24:20.940 --> 00:24:22.800 align:middle line:84% Now, you can simplify that further 00:24:22.800 --> 00:24:24.630 align:middle line:84% for the camera world, where we're 00:24:24.630 --> 00:24:26.788 align:middle line:84% going to assign the plane of a sensor, 00:24:26.788 --> 00:24:28.330 align:middle line:84% and the plane of the lens, and so on. 00:24:28.330 --> 00:24:31.930 align:middle line:84% So that's what we'll see very briefly. 00:24:31.930 --> 00:24:33.930 align:middle line:84% So let's say there's a light field in this room. 00:24:33.930 --> 00:24:36.510 align:middle line:84% Rays are traveling from light sources, 00:24:36.510 --> 00:24:37.800 align:middle line:90% bouncing around everywhere. 00:24:37.800 --> 00:24:44.130 align:middle line:84% If I just cut a plane in midair, I can parameterize that plane 00:24:44.130 --> 00:24:46.330 align:middle line:90% as x- and y-coordinates. 00:24:46.330 --> 00:24:48.330 align:middle line:84% And for every point on the plane, 00:24:48.330 --> 00:24:51.000 align:middle line:84% I have, again, the theta and phi. 00:24:51.000 --> 00:24:52.560 align:middle line:84% So this becomes four dimensional. 00:24:52.560 --> 00:24:54.492 align:middle line:90% 00:24:54.492 --> 00:24:55.950 align:middle line:84% And that's what we're showing here. 00:24:55.950 --> 00:24:58.710 align:middle line:84% The positions is s and the direction is theta. 00:24:58.710 --> 00:25:01.950 align:middle line:84% So often, we will think about flat land. 00:25:01.950 --> 00:25:05.550 align:middle line:84% So we'll just think about the plane of the screen as opposed 00:25:05.550 --> 00:25:06.610 align:middle line:90% to the 3D world. 00:25:06.610 --> 00:25:10.000 align:middle line:84% So in the 3D world, we have x, y, and theta phi. 00:25:10.000 --> 00:25:14.880 align:middle line:84% But in flat land, we have just the position and angle. 00:25:14.880 --> 00:25:16.134 align:middle line:90% So it's just two dimensional. 00:25:16.134 --> 00:25:19.040 align:middle line:90% 00:25:19.040 --> 00:25:21.880 align:middle line:90% So this is called a-- 00:25:21.880 --> 00:25:23.980 align:middle line:84% so that was one-plane parameterization, where 00:25:23.980 --> 00:25:25.480 align:middle line:90% you had position and angle. 00:25:25.480 --> 00:25:29.080 align:middle line:84% And another common way to think about that is-- 00:25:29.080 --> 00:25:31.570 align:middle line:84% another common way to parameterize the light field 00:25:31.570 --> 00:25:34.810 align:middle line:84% is a two-plane parameterization, where 00:25:34.810 --> 00:25:37.420 align:middle line:84% you have one plane that has position, 00:25:37.420 --> 00:25:39.850 align:middle line:84% and the second plane that, again, has position, 00:25:39.850 --> 00:25:43.690 align:middle line:84% and a ray that connects those two, again, 00:25:43.690 --> 00:25:47.330 align:middle line:90% represents the ray space. 00:25:47.330 --> 00:25:50.090 align:middle line:84% The coordinates for that represent the ray space. 00:25:50.090 --> 00:25:53.420 align:middle line:84% So this is so far two-plane parameterization. 00:25:53.420 --> 00:25:57.620 align:middle line:84% And this is very commonly used in computational camera 00:25:57.620 --> 00:25:58.882 align:middle line:90% and photography. 00:25:58.882 --> 00:26:02.320 align:middle line:90% 00:26:02.320 --> 00:26:05.950 align:middle line:84% So let me jump ahead a little bit because of the time left 00:26:05.950 --> 00:26:09.870 align:middle line:84% and explain how we're going to do it for our assignment. 00:26:09.870 --> 00:26:13.300 align:middle line:90% 00:26:13.300 --> 00:26:17.790 align:middle line:84% So remember, we're going to create an effect where we'll 00:26:17.790 --> 00:26:20.610 align:middle line:84% put a whole bunch of cameras or take an array of cameras 00:26:20.610 --> 00:26:25.980 align:middle line:84% like this and be able to see 12 [? rulers. ?] 00:26:25.980 --> 00:26:28.534 align:middle line:84% And the effect is relatively straightforward. 00:26:28.534 --> 00:26:32.410 align:middle line:90% 00:26:32.410 --> 00:26:36.420 align:middle line:84% And we go to the so-called synthetic aperture photography. 00:26:36.420 --> 00:26:38.880 align:middle line:84% We're going to create an artificial aperture 00:26:38.880 --> 00:26:45.330 align:middle line:84% to be able to see through [INAUDIBLE].. 00:26:45.330 --> 00:26:48.570 align:middle line:84% So if you have a point [INAUDIBLE] focus 00:26:48.570 --> 00:26:52.380 align:middle line:84% versus a point that's out of focus, 00:26:52.380 --> 00:26:57.720 align:middle line:84% the green point will create a very bright spot. 00:26:57.720 --> 00:27:00.060 align:middle line:84% The red point will create a blurred spot. 00:27:00.060 --> 00:27:03.120 align:middle line:84% That means its intensity will be correspondingly reduced 00:27:03.120 --> 00:27:06.220 align:middle line:90% per pixel. 00:27:06.220 --> 00:27:09.020 align:middle line:84% And if you stopped on the picture, what will happen 00:27:09.020 --> 00:27:13.280 align:middle line:84% is that the green spot will become slightly dimmer, 00:27:13.280 --> 00:27:15.350 align:middle line:84% because less light is reaching the sensor. 00:27:15.350 --> 00:27:18.050 align:middle line:84% But the red spot will also focus-- 00:27:18.050 --> 00:27:21.610 align:middle line:90% also blur in a smaller region. 00:27:21.610 --> 00:27:24.200 align:middle line:84% If you go in the opposite direction and have a really, 00:27:24.200 --> 00:27:28.810 align:middle line:84% really large aperture, then the green spot will be very bright, 00:27:28.810 --> 00:27:31.453 align:middle line:84% because it's capturing-- a lot of light is being captured-- 00:27:31.453 --> 00:27:32.620 align:middle line:90% and that will be over here-- 00:27:32.620 --> 00:27:34.900 align:middle line:84% but the red spot will be highly blurred. 00:27:34.900 --> 00:27:38.940 align:middle line:90% 00:27:38.940 --> 00:27:41.770 align:middle line:84% Now, building such a large aperture is very challenging. 00:27:41.770 --> 00:27:44.940 align:middle line:84% So what we're going to do is create that using 00:27:44.940 --> 00:27:46.965 align:middle line:90% an array of cameras, like this. 00:27:46.965 --> 00:27:49.710 align:middle line:90% 00:27:49.710 --> 00:27:51.890 align:middle line:84% And it's the same as synthetic aperture radar, 00:27:51.890 --> 00:27:55.460 align:middle line:84% where they use an array of antennas to create, 00:27:55.460 --> 00:27:57.500 align:middle line:84% effectively, a much larger antenna. 00:27:57.500 --> 00:28:00.590 align:middle line:84% Again, analogies with communication, and RF, and so 00:28:00.590 --> 00:28:02.970 align:middle line:90% on. 00:28:02.970 --> 00:28:07.620 align:middle line:90% So again, the whole point that-- 00:28:07.620 --> 00:28:10.550 align:middle line:84% so again, we're going to subdivide this lens 00:28:10.550 --> 00:28:15.830 align:middle line:84% into multiple apertures as opposed to one large aperture. 00:28:15.830 --> 00:28:17.660 align:middle line:84% And that will be effectively created 00:28:17.660 --> 00:28:19.850 align:middle line:90% with a set of cameras like this. 00:28:19.850 --> 00:28:23.120 align:middle line:84% And then if you sum the images from each of those apertures, 00:28:23.120 --> 00:28:25.280 align:middle line:84% that's the same as creating an image 00:28:25.280 --> 00:28:26.600 align:middle line:90% with this very large lens. 00:28:26.600 --> 00:28:30.110 align:middle line:90% 00:28:30.110 --> 00:28:34.760 align:middle line:84% And for a different point in 3D, will correspond and create 00:28:34.760 --> 00:28:36.424 align:middle line:90% a different image. 00:28:36.424 --> 00:28:37.368 align:middle line:90% That's it. 00:28:37.368 --> 00:28:40.200 align:middle line:90% 00:28:40.200 --> 00:28:42.070 align:middle line:90% [INAUDIBLE] 00:28:42.070 --> 00:28:42.570 align:middle line:90% OK. 00:28:42.570 --> 00:28:43.510 align:middle line:90% So how does this work? 00:28:43.510 --> 00:28:48.510 align:middle line:84% How are we going to create an effect where 00:28:48.510 --> 00:28:52.740 align:middle line:84% something that's out of focus effectively 00:28:52.740 --> 00:28:54.330 align:middle line:84% is going to be completely blurred? 00:28:54.330 --> 00:28:57.360 align:middle line:90% And we saw that if-- 00:28:57.360 --> 00:29:00.420 align:middle line:84% even with aperture of my eye, which 00:29:00.420 --> 00:29:03.360 align:middle line:84% is only about 6 millimeters or so, if I put an object really 00:29:03.360 --> 00:29:08.340 align:middle line:84% close, then I can see the world through this eye 00:29:08.340 --> 00:29:11.520 align:middle line:84% so that this is basically-- it doesn't impact. 00:29:11.520 --> 00:29:15.000 align:middle line:84% If I put a needle in front of me, it gets completely blurred. 00:29:15.000 --> 00:29:18.660 align:middle line:84% And that's the same effect we're going to see. 00:29:18.660 --> 00:29:22.310 align:middle line:84% So you take an array of cameras or camera 00:29:22.310 --> 00:29:26.010 align:middle line:90% at different positions, take-- 00:29:26.010 --> 00:29:28.530 align:middle line:90% collect, say, 25 photos. 00:29:28.530 --> 00:29:34.302 align:middle line:84% If you simply take those 25 photos and sum them up, 00:29:34.302 --> 00:29:35.010 align:middle line:90% what will happen? 00:29:35.010 --> 00:29:38.270 align:middle line:90% 00:29:38.270 --> 00:29:40.370 align:middle line:90% So if I just take a camera-- 00:29:40.370 --> 00:29:43.320 align:middle line:90% 00:29:43.320 --> 00:29:45.734 align:middle line:84% and for simplicity, we're just going to [INAUDIBLE].. 00:29:45.734 --> 00:29:50.880 align:middle line:90% 00:29:50.880 --> 00:29:53.110 align:middle line:84% If I have, say, five cameras this way, 00:29:53.110 --> 00:29:56.230 align:middle line:84% and I will point at the [INAUDIBLE],, 00:29:56.230 --> 00:29:57.903 align:middle line:84% this coordinate in each of these cameras 00:29:57.903 --> 00:29:59.320 align:middle line:84% is going to be added [INAUDIBLE],, 00:29:59.320 --> 00:30:00.700 align:middle line:90% because it's [INAUDIBLE]. 00:30:00.700 --> 00:30:04.150 align:middle line:84% It's like when you're driving and you're looking at the moon, 00:30:04.150 --> 00:30:07.366 align:middle line:84% it always appears to be the same position. 00:30:07.366 --> 00:30:10.136 align:middle line:84% If you're looking at a [INAUDIBLE] which is very, very 00:30:10.136 --> 00:30:12.800 align:middle line:84% far away, its coordinate in the camera 00:30:12.800 --> 00:30:17.450 align:middle line:84% is going to be the same [INAUDIBLE].. 00:30:17.450 --> 00:30:19.570 align:middle line:84% So if I just take these five photos 00:30:19.570 --> 00:30:23.320 align:middle line:84% and add them up, sum them up, I basically 00:30:23.320 --> 00:30:27.310 align:middle line:84% get the same exact [INAUDIBLE],, because I [INAUDIBLE].. 00:30:27.310 --> 00:30:30.820 align:middle line:84% Now let me [INAUDIBLE] something that's nearby. 00:30:30.820 --> 00:30:33.970 align:middle line:84% This coordinate in the top camera 00:30:33.970 --> 00:30:37.420 align:middle line:84% will be the top of the point of infinity. 00:30:37.420 --> 00:30:41.110 align:middle line:84% But in the bottom camera, it will 00:30:41.110 --> 00:30:44.420 align:middle line:90% be below the point of infinity. 00:30:44.420 --> 00:30:48.010 align:middle line:84% So now you have summed these five images up, 00:30:48.010 --> 00:30:51.177 align:middle line:84% this point will be completely blurred, 00:30:51.177 --> 00:30:53.260 align:middle line:84% because the coordinates in each of the five images 00:30:53.260 --> 00:30:55.030 align:middle line:90% are very different. 00:30:55.030 --> 00:30:56.860 align:middle line:84% On the other hand, what's at infinity 00:30:56.860 --> 00:30:58.660 align:middle line:90% will be the [INAUDIBLE]. 00:30:58.660 --> 00:31:01.000 align:middle line:84% And that's exactly what's happening when you focus. 00:31:01.000 --> 00:31:03.770 align:middle line:84% You're basically taking an image from every part of the lens 00:31:03.770 --> 00:31:05.800 align:middle line:90% and summing it up [INAUDIBLE]. 00:31:05.800 --> 00:31:08.530 align:middle line:90% Here we're doing it in software. 00:31:08.530 --> 00:31:10.540 align:middle line:84% And of course, mathematically we're 00:31:10.540 --> 00:31:12.890 align:middle line:90% not going to sum it up as is. 00:31:12.890 --> 00:31:15.440 align:middle line:84% We can shift each image and sum it. 00:31:15.440 --> 00:31:18.280 align:middle line:84% So if I wanted to focus here, what I would do 00:31:18.280 --> 00:31:21.797 align:middle line:84% is I would take this picture, keep it as is, 00:31:21.797 --> 00:31:22.880 align:middle line:90% and I'd take this picture. 00:31:22.880 --> 00:31:26.710 align:middle line:84% And I know that from here to here, 00:31:26.710 --> 00:31:28.750 align:middle line:84% this one is shifting left about 5 pixels. 00:31:28.750 --> 00:31:32.360 align:middle line:84% I'll shift this whole image by 5 pixels and then add it. 00:31:32.360 --> 00:31:33.370 align:middle line:90% I'll take the next one. 00:31:33.370 --> 00:31:35.828 align:middle line:84% And I know that's going to be about 10 pixels [INAUDIBLE].. 00:31:35.828 --> 00:31:38.475 align:middle line:84% I'll shift the whole image by 10 pixels and then add it. 00:31:38.475 --> 00:31:41.230 align:middle line:84% And the bottom one, I have to shift by 20 pixels 00:31:41.230 --> 00:31:42.230 align:middle line:90% and then add it. 00:31:42.230 --> 00:31:45.140 align:middle line:84% If I do that, then this point will be sharp focus. 00:31:45.140 --> 00:31:49.180 align:middle line:84% And the point at infinity will be completely blurred. 00:31:49.180 --> 00:31:53.900 align:middle line:84% And using this very simple shift and add mechanism, 00:31:53.900 --> 00:31:56.390 align:middle line:84% we're able to focus very close [INAUDIBLE].. 00:31:56.390 --> 00:31:59.120 align:middle line:90% 00:31:59.120 --> 00:32:01.315 align:middle line:90% Is this clear? 00:32:01.315 --> 00:32:03.190 align:middle line:84% AUDIENCE: I have a quick question about this. 00:32:03.190 --> 00:32:03.857 align:middle line:90% PROFESSOR: Yeah. 00:32:03.857 --> 00:32:07.927 align:middle line:84% AUDIENCE: Does this set some minimum focus distance? 00:32:07.927 --> 00:32:08.760 align:middle line:90% PROFESSOR: It could. 00:32:08.760 --> 00:32:09.120 align:middle line:90% It could, yes 00:32:09.120 --> 00:32:11.290 align:middle line:84% AUDIENCE: Or what [INAUDIBLE] minimum focus distance 00:32:11.290 --> 00:32:11.920 align:middle line:90% does it need? 00:32:11.920 --> 00:32:14.003 align:middle line:84% PROFESSOR: The field of view is set-- for example, 00:32:14.003 --> 00:32:16.540 align:middle line:84% if I get really close, then these cameras can 00:32:16.540 --> 00:32:19.480 align:middle line:90% see the sky, [INAUDIBLE]. 00:32:19.480 --> 00:32:21.687 align:middle line:90% So that's [INAUDIBLE]. 00:32:21.687 --> 00:32:23.520 align:middle line:84% AUDIENCE: So is that the only thing, though? 00:32:23.520 --> 00:32:26.120 align:middle line:84% Is it just the field of view that [INAUDIBLE] focal point? 00:32:26.120 --> 00:32:27.620 align:middle line:84% PROFESSOR: Mainly the field of view. 00:32:27.620 --> 00:32:29.750 align:middle line:84% The resolution [INAUDIBLE] as well. 00:32:29.750 --> 00:32:31.910 align:middle line:90% But mainly the field of view. 00:32:31.910 --> 00:32:35.450 align:middle line:84% Otherwise, this technique can focus on [INAUDIBLE].. 00:32:35.450 --> 00:32:38.265 align:middle line:84% It can even focus beyond infinity. 00:32:38.265 --> 00:32:40.264 align:middle line:90% So first [INAUDIBLE]. 00:32:40.264 --> 00:32:44.340 align:middle line:90% [LAUGHTER] 00:32:44.340 --> 00:32:46.340 align:middle line:84% Because you can-- you're sort of adding them up. 00:32:46.340 --> 00:32:50.190 align:middle line:84% I can add them up in the reverse direction, minus 5 pixels, 00:32:50.190 --> 00:32:52.310 align:middle line:90% and focusing at infinity beyond. 00:32:52.310 --> 00:32:56.670 align:middle line:90% 00:32:56.670 --> 00:32:58.690 align:middle line:84% So this is all we're going to do. 00:32:58.690 --> 00:33:01.850 align:middle line:84% But that's-- as you'll realize in your assignment, 00:33:01.850 --> 00:33:04.131 align:middle line:84% there are a few things [? you ?] have learned. 00:33:04.131 --> 00:33:06.960 align:middle line:84% Here, I just choose some numbers, and shift by 5 pixels, 00:33:06.960 --> 00:33:07.460 align:middle line:90% and add. 00:33:07.460 --> 00:33:10.560 align:middle line:84% What will end up happening in your case is you'll realize 00:33:10.560 --> 00:33:15.990 align:middle line:84% that if you put these cameras and take the pictures and then 00:33:15.990 --> 00:33:18.570 align:middle line:84% figure out what distance this should be, 00:33:18.570 --> 00:33:21.880 align:middle line:84% what the projection of these points are going to be, 00:33:21.880 --> 00:33:26.760 align:middle line:84% and if you don't use some meaningful numbers, 00:33:26.760 --> 00:33:30.840 align:middle line:84% you'll never get [INAUDIBLE],, because either your parallax, 00:33:30.840 --> 00:33:33.780 align:middle line:84% which is the distance-- the change in coordinates as you 00:33:33.780 --> 00:33:35.700 align:middle line:84% switch from one [INAUDIBLE] to the other-- 00:33:35.700 --> 00:33:36.930 align:middle line:90% too large or too small. 00:33:36.930 --> 00:33:40.770 align:middle line:90% 00:33:40.770 --> 00:33:42.750 align:middle line:84% And it's really easy to do by just kind 00:33:42.750 --> 00:33:45.160 align:middle line:84% of eyeballing it, but eyeballing it 00:33:45.160 --> 00:33:47.640 align:middle line:84% not with camera, with your own eyes. 00:33:47.640 --> 00:33:49.620 align:middle line:84% So you can just stand in one place 00:33:49.620 --> 00:33:52.890 align:middle line:84% and see if you move by 10 centimeters, 00:33:52.890 --> 00:33:56.380 align:middle line:84% do you eventually see the point behind? 00:33:56.380 --> 00:33:59.700 align:middle line:84% And in the case of the Stanford project, 00:33:59.700 --> 00:34:01.290 align:middle line:84% that was a really challenging example 00:34:01.290 --> 00:34:03.368 align:middle line:84% of a set of trees and people behind them. 00:34:03.368 --> 00:34:05.535 align:middle line:84% You don't have to choose something that complicated. 00:34:05.535 --> 00:34:09.179 align:middle line:84% You can choose some set of objects in the front 00:34:09.179 --> 00:34:12.239 align:middle line:84% and then some painting in the background. 00:34:12.239 --> 00:34:15.275 align:middle line:84% If you want to set up the scene, and you want to put some-- 00:34:15.275 --> 00:34:16.650 align:middle line:84% the best would be really just put 00:34:16.650 --> 00:34:21.929 align:middle line:84% a pencil first in the [INAUDIBLE] to a [? fence, ?] 00:34:21.929 --> 00:34:25.500 align:middle line:84% and then there's a painting in the background. 00:34:25.500 --> 00:34:27.810 align:middle line:84% And then if from any single camera 00:34:27.810 --> 00:34:32.710 align:middle line:84% this painting is occluded, by taking multiple photos, 00:34:32.710 --> 00:34:34.290 align:middle line:90% you can see-- 00:34:34.290 --> 00:34:36.570 align:middle line:90% I can do it on a table. 00:34:36.570 --> 00:34:39.386 align:middle line:84% You can do it outdoors, for example. 00:34:39.386 --> 00:34:40.469 align:middle line:90% There are some trees here. 00:34:40.469 --> 00:34:42.330 align:middle line:90% You can see through it. 00:34:42.330 --> 00:34:48.002 align:middle line:84% Choose your situation, and you'll be able to answer this. 00:34:48.002 --> 00:34:51.420 align:middle line:90% 00:34:51.420 --> 00:34:53.730 align:middle line:84% And there will be more instructions on the website. 00:34:53.730 --> 00:34:56.558 align:middle line:84% AUDIENCE: Are we allowed to do it all computationally? 00:34:56.558 --> 00:34:57.600 align:middle line:90% PROFESSOR: Pure software? 00:34:57.600 --> 00:34:57.960 align:middle line:90% AUDIENCE: Yeah. 00:34:57.960 --> 00:34:58.720 align:middle line:84% PROFESSOR: Just OpenGL, you mean? 00:34:58.720 --> 00:34:59.470 align:middle line:90% Absolutely, yeah. 00:34:59.470 --> 00:34:59.970 align:middle line:90% Yeah. 00:34:59.970 --> 00:35:02.970 align:middle line:84% But I mean, where's the fun that? 00:35:02.970 --> 00:35:04.940 align:middle line:84% You are perfectly welcome to do that, yeah. 00:35:04.940 --> 00:35:10.050 align:middle line:84% AUDIENCE: What's the tolerance for the parallelism? 00:35:10.050 --> 00:35:13.620 align:middle line:84% PROFESSOR: So you want to be as close to parallel as possible. 00:35:13.620 --> 00:35:16.150 align:middle line:84% But-- we are not going to discuss it here, 00:35:16.150 --> 00:35:18.070 align:middle line:84% but as you know that once you have-- 00:35:18.070 --> 00:35:20.450 align:middle line:84% for example, if you misalign this camera 00:35:20.450 --> 00:35:24.380 align:middle line:84% and collect your [INAUDIBLE],, then you 00:35:24.380 --> 00:35:26.580 align:middle line:84% know that this image to this image 00:35:26.580 --> 00:35:30.962 align:middle line:84% is just a pure one model, a pure, single [INAUDIBLE].. 00:35:30.962 --> 00:35:32.670 align:middle line:84% So you could just fix that mathematically 00:35:32.670 --> 00:35:35.100 align:middle line:90% if you wanted to. 00:35:35.100 --> 00:35:37.710 align:middle line:84% But you should just avoid that for this assignment. 00:35:37.710 --> 00:35:38.940 align:middle line:90% Try to keep it parallel. 00:35:38.940 --> 00:35:41.140 align:middle line:84% Just put it on a ruler and slide it. 00:35:41.140 --> 00:35:47.270 align:middle line:90% 00:35:47.270 --> 00:35:51.830 align:middle line:84% And I'll give-- there's more information on the website. 00:35:51.830 --> 00:35:53.287 align:middle line:90%