WEBVTT 00:00:00.000 --> 00:00:02.490 align:middle line:84% The following content is provided under a Creative 00:00:02.490 --> 00:00:03.730 align:middle line:90% Commons license. 00:00:03.730 --> 00:00:06.120 align:middle line:84% Your support will help MIT OpenCourseWare 00:00:06.120 --> 00:00:10.090 align:middle line:84% continue to offer high quality educational resources for free. 00:00:10.090 --> 00:00:12.720 align:middle line:84% To make a donation or to view additional materials 00:00:12.720 --> 00:00:16.650 align:middle line:84% from hundreds of MIT courses, visit MIT OpenCourseWare 00:00:16.650 --> 00:00:17.980 align:middle line:90% at ocw.mit.edu. 00:00:17.980 --> 00:00:20.840 align:middle line:90% 00:00:20.840 --> 00:00:23.580 align:middle line:84% RAMESH RASKAR: All right, everyone, let's get started. 00:00:23.580 --> 00:00:24.720 align:middle line:90% I am Ramesh Raskar. 00:00:24.720 --> 00:00:26.540 align:middle line:90% And this is-- 00:00:26.540 --> 00:00:29.490 align:middle line:90% AUDIENCE: I don't know. 00:00:29.490 --> 00:00:33.690 align:middle line:84% RAMESH RASKAR: --MAS 131 and 531, computational camera 00:00:33.690 --> 00:00:35.370 align:middle line:90% and photography. 00:00:35.370 --> 00:00:40.170 align:middle line:84% And we have, I think, about $50 to $60,000 worth of equipment 00:00:40.170 --> 00:00:41.177 align:middle line:90% right here. 00:00:41.177 --> 00:00:43.260 align:middle line:84% So please hold on to the door if something goes on 00:00:43.260 --> 00:00:46.320 align:middle line:84% I really need your help to protect all these things. 00:00:46.320 --> 00:00:48.130 align:middle line:90% It's going to be a lot of fun. 00:00:48.130 --> 00:00:50.340 align:middle line:84% We're going to see crazy kinds of cameras, 00:00:50.340 --> 00:00:52.860 align:middle line:84% crazy kind of photography, medical imaging, 00:00:52.860 --> 00:00:55.890 align:middle line:84% and applications in different domains. 00:00:55.890 --> 00:00:58.200 align:middle line:84% I just wanted to show this really beautiful picture 00:00:58.200 --> 00:01:03.180 align:middle line:84% that some of you may have seen on blogs. 00:01:03.180 --> 00:01:09.510 align:middle line:84% This is a real photo taken by an iPhone camera of a propeller 00:01:09.510 --> 00:01:12.630 align:middle line:90% blades on an aircraft. 00:01:12.630 --> 00:01:15.960 align:middle line:90% Anybody knows what's going on? 00:01:15.960 --> 00:01:18.300 align:middle line:84% AUDIENCE: The sensor takes line by line images. 00:01:18.300 --> 00:01:21.105 align:middle line:84% And by the time it goes to the next line, next vertical line, 00:01:21.105 --> 00:01:22.560 align:middle line:90% those propellers have moved. 00:01:22.560 --> 00:01:26.700 align:middle line:84% So it faces the [INAUDIBLE] directions are not exact 00:01:26.700 --> 00:01:27.200 align:middle line:90% location-- 00:01:27.200 --> 00:01:28.242 align:middle line:90% RAMESH RASKAR: Excellent. 00:01:28.242 --> 00:01:30.300 align:middle line:84% AUDIENCE: [INAUDIBLE] separated [INAUDIBLE].. 00:01:30.300 --> 00:01:32.633 align:middle line:84% RAMESH RASKAR: Exactly, so when we think about an image, 00:01:32.633 --> 00:01:35.700 align:middle line:84% we think about this as some kind of a snapshot, some kind 00:01:35.700 --> 00:01:37.320 align:middle line:90% of a progressive photo. 00:01:37.320 --> 00:01:39.520 align:middle line:84% But there are two motions going on here. 00:01:39.520 --> 00:01:43.950 align:middle line:84% One is the motion of the blade, which is circular, 00:01:43.950 --> 00:01:45.810 align:middle line:90% and with radial blades. 00:01:45.810 --> 00:01:50.520 align:middle line:84% And then there is motion effectively 00:01:50.520 --> 00:01:53.340 align:middle line:84% of the sampler, which is moving from up to down. 00:01:53.340 --> 00:01:56.460 align:middle line:90% It's a rolling shutter. 00:01:56.460 --> 00:02:00.330 align:middle line:84% There's a very nice animation of that. 00:02:00.330 --> 00:02:04.050 align:middle line:90% 00:02:04.050 --> 00:02:06.320 align:middle line:84% So this is the rolling shutter the camera 00:02:06.320 --> 00:02:09.900 align:middle line:84% is exposing approximately just one line at a time. 00:02:09.900 --> 00:02:13.400 align:middle line:84% And as the blades rotate around, you 00:02:13.400 --> 00:02:17.210 align:middle line:84% can see that it's tracing this course. 00:02:17.210 --> 00:02:22.190 align:middle line:84% So this example should show you that we can't take anything 00:02:22.190 --> 00:02:28.430 align:middle line:84% for granted when it comes to this modern photography. 00:02:28.430 --> 00:02:31.880 align:middle line:84% And this is mostly an artifact of a really cheap sensor 00:02:31.880 --> 00:02:33.290 align:middle line:90% that cell phone cameras use. 00:02:33.290 --> 00:02:35.750 align:middle line:84% Because of bandwidth constraints it's much easier 00:02:35.750 --> 00:02:37.940 align:middle line:84% to roll and read one line at a time 00:02:37.940 --> 00:02:40.850 align:middle line:84% than reading the whole buffer of the camera at the same time. 00:02:40.850 --> 00:02:42.890 align:middle line:84% So you get just beautiful artifacts. 00:02:42.890 --> 00:02:47.120 align:middle line:84% And one open question is if camera makers 00:02:47.120 --> 00:02:51.530 align:middle line:84% start supporting or start cheaping out on 00:02:51.530 --> 00:02:53.630 align:middle line:84% rather than just one line at a time-- 00:02:53.630 --> 00:02:56.270 align:middle line:84% if they come up with even cheaper mechanisms 00:02:56.270 --> 00:03:00.380 align:middle line:84% where they're explaining and some random sequence or-- 00:03:00.380 --> 00:03:03.590 align:middle line:84% any artifacts they create, hover photographers 00:03:03.590 --> 00:03:09.470 align:middle line:84% exploit that to create this stunning and beautiful imagery. 00:03:09.470 --> 00:03:11.920 align:middle line:84% More on the scientific side, let's see the lighting 00:03:11.920 --> 00:03:14.000 align:middle line:90% is [INAUDIBLE] here. 00:03:14.000 --> 00:03:16.560 align:middle line:84% Let's see if you can play with that. 00:03:16.560 --> 00:03:18.350 align:middle line:90% Yeah, I'll do that. 00:03:18.350 --> 00:03:21.210 align:middle line:84% See if there's another switch that doesn't like the back-- 00:03:21.210 --> 00:03:21.710 align:middle line:90% No. 00:03:21.710 --> 00:03:24.650 align:middle line:90% 00:03:24.650 --> 00:03:27.590 align:middle line:84% We're going to learn a lot about computational illumination 00:03:27.590 --> 00:03:28.683 align:middle line:90% in this [INAUDIBLE]. 00:03:28.683 --> 00:03:29.600 align:middle line:90% That maybe OK for now. 00:03:29.600 --> 00:03:31.770 align:middle line:90% And we'll switch it back. 00:03:31.770 --> 00:03:38.030 align:middle line:84% So imagine you have somebody out here behind a shower curtain 00:03:38.030 --> 00:03:39.808 align:middle line:90% and you want to take a photo. 00:03:39.808 --> 00:03:41.600 align:middle line:84% And you may be able to come up with a photo 00:03:41.600 --> 00:03:44.300 align:middle line:84% where you just see the shower curtain behind the person 00:03:44.300 --> 00:03:49.460 align:middle line:84% behind without the person behind the shower curtain. 00:03:49.460 --> 00:03:51.590 align:middle line:84% But maybe you can also create a photo 00:03:51.590 --> 00:03:57.390 align:middle line:84% where you realize it was behind the shower curtain. 00:03:57.390 --> 00:04:01.320 align:middle line:84% And this is a trick that's achieved using 00:04:01.320 --> 00:04:03.360 align:middle line:90% a special type of a flash. 00:04:03.360 --> 00:04:05.940 align:middle line:84% And as you'll see, we will generalize the flash 00:04:05.940 --> 00:04:09.840 align:middle line:84% to a very highly programmable illumination. 00:04:09.840 --> 00:04:13.650 align:middle line:84% And with that, photography of this kind is also possible. 00:04:13.650 --> 00:04:15.040 align:middle line:90% What else? 00:04:15.040 --> 00:04:18.029 align:middle line:84% Here's a great example I'd like to show. 00:04:18.029 --> 00:04:21.300 align:middle line:84% Imagine you have a scene and instead of a camera 00:04:21.300 --> 00:04:24.020 align:middle line:84% you just have a single photodetector. 00:04:24.020 --> 00:04:28.710 align:middle line:84% OK, It could be a light sensor of your SLR camera. 00:04:28.710 --> 00:04:32.130 align:middle line:84% Those of you like to think about these photodiodes, just one 00:04:32.130 --> 00:04:32.850 align:middle line:90% photodiode. 00:04:32.850 --> 00:04:35.640 align:middle line:84% And instead of taking a photo, what I'm going to do 00:04:35.640 --> 00:04:38.820 align:middle line:84% is turn on one pixel of this projector at a time 00:04:38.820 --> 00:04:40.290 align:middle line:90% and record this. 00:04:40.290 --> 00:04:42.570 align:middle line:84% If the projector has one million pixels, 00:04:42.570 --> 00:04:44.760 align:middle line:84% I'm going to take one million measurements 00:04:44.760 --> 00:04:46.530 align:middle line:90% by turning on one at a time. 00:04:46.530 --> 00:04:48.120 align:middle line:90% What picture will I get here? 00:04:48.120 --> 00:04:50.760 align:middle line:90% 00:04:50.760 --> 00:04:55.212 align:middle line:84% Will I get a picture from those 1 million measurements? 00:04:55.212 --> 00:04:57.780 align:middle line:90% Anybody? 00:04:57.780 --> 00:05:01.260 align:middle line:84% No, you get a photo that looks like the camera was placed 00:05:01.260 --> 00:05:04.710 align:middle line:90% here instead of over here. 00:05:04.710 --> 00:05:08.267 align:middle line:84% And it's exactly how a barcode scanner at checkout counter 00:05:08.267 --> 00:05:08.850 align:middle line:90% works as well. 00:05:08.850 --> 00:05:11.200 align:middle line:84% The barcode scanner doesn't have a camera. 00:05:11.200 --> 00:05:14.190 align:middle line:90% It just has a sweeping laser. 00:05:14.190 --> 00:05:17.010 align:middle line:84% In the same apparatus there's also a single photodetector. 00:05:17.010 --> 00:05:20.430 align:middle line:84% So if the barcode scanner hits the 1010 patterns 00:05:20.430 --> 00:05:24.210 align:middle line:84% of the printed code, when it hits the black spot, 00:05:24.210 --> 00:05:26.760 align:middle line:84% it doesn't reflect much light and it hits the white spot. 00:05:26.760 --> 00:05:28.110 align:middle line:90% It reflects much light. 00:05:28.110 --> 00:05:30.810 align:middle line:84% And in this way, a barcode scanner 00:05:30.810 --> 00:05:34.230 align:middle line:84% without having to worry about focus and dynamic range and all 00:05:34.230 --> 00:05:37.038 align:middle line:84% those issues can figure out what the barcode is. 00:05:37.038 --> 00:05:38.580 align:middle line:84% AUDIENCE: Isn't it important, though, 00:05:38.580 --> 00:05:40.770 align:middle line:84% that the barcode scanner is a laser, which keeps 00:05:40.770 --> 00:05:42.783 align:middle line:90% the pixels spatially localized? 00:05:42.783 --> 00:05:43.950 align:middle line:90% RAMESH RASKAR: Very focused. 00:05:43.950 --> 00:05:44.617 align:middle line:90% AUDIENCE: Right. 00:05:44.617 --> 00:05:47.160 align:middle line:84% Whereas here if you have an incoherent light source, 00:05:47.160 --> 00:05:49.020 align:middle line:84% and you're illuminating a real scene, 00:05:49.020 --> 00:05:51.263 align:middle line:84% you'll illuminate the whole scene kind of diffusely. 00:05:51.263 --> 00:05:52.930 align:middle line:84% RAMESH RASKAR: That's a very good point. 00:05:52.930 --> 00:05:54.638 align:middle line:84% So that's why we have to use a projector. 00:05:54.638 --> 00:05:57.120 align:middle line:84% So we are turning on only one pixel of the projector. 00:05:57.120 --> 00:05:59.760 align:middle line:84% So as in very simple words, it's turning 00:05:59.760 --> 00:06:02.550 align:middle line:90% on only one ray in the scene. 00:06:02.550 --> 00:06:04.590 align:middle line:90% The projector is not flat field. 00:06:04.590 --> 00:06:07.140 align:middle line:84% Not all the pixels are on it at a time. 00:06:07.140 --> 00:06:10.590 align:middle line:84% That's why we had to take one million readings while turning 00:06:10.590 --> 00:06:13.560 align:middle line:90% on one pixel at a time. 00:06:13.560 --> 00:06:15.040 align:middle line:90% So that's straightforward. 00:06:15.040 --> 00:06:17.250 align:middle line:90% This is very well known. 00:06:17.250 --> 00:06:19.050 align:middle line:84% What are some other things you can do 00:06:19.050 --> 00:06:22.650 align:middle line:90% with this particular duality? 00:06:22.650 --> 00:06:26.310 align:middle line:84% If you replace the photodetector with a camera, 00:06:26.310 --> 00:06:27.540 align:middle line:90% you can do something similar. 00:06:27.540 --> 00:06:31.000 align:middle line:84% You can turn on one pixel at a time and take a full photo. 00:06:31.000 --> 00:06:32.970 align:middle line:84% So again, I'm going to turn on one 00:06:32.970 --> 00:06:35.970 align:middle line:84% pixel of the projector at a time and in this case, 00:06:35.970 --> 00:06:40.080 align:middle line:90% take 1 million photos. 00:06:40.080 --> 00:06:42.930 align:middle line:84% Now, once you do that, what I will be able to do 00:06:42.930 --> 00:06:45.060 align:middle line:84% is create a relationship between what 00:06:45.060 --> 00:06:49.830 align:middle line:84% happens when I turn on exactly one pixel of this projector. 00:06:49.830 --> 00:06:51.060 align:middle line:90% One pixel is turned on. 00:06:51.060 --> 00:06:51.990 align:middle line:90% A photo is taken. 00:06:51.990 --> 00:06:54.450 align:middle line:84% I can measure what happens to this particular pixel 00:06:54.450 --> 00:06:55.930 align:middle line:90% of the camera. 00:06:55.930 --> 00:06:58.830 align:middle line:84% And in this way, you will create a four-dimensional 00:06:58.830 --> 00:07:05.550 align:middle line:84% relationship, 2D for the camera and 2D for the-- 00:07:05.550 --> 00:07:07.930 align:middle line:84% so 2D for the projector and 2D for the camera. 00:07:07.930 --> 00:07:11.100 align:middle line:84% So a million photos here and a million pixels here. 00:07:11.100 --> 00:07:15.510 align:middle line:84% So you're going to have a trillion measurements, 10 00:07:15.510 --> 00:07:17.550 align:middle line:90% to the 12. 00:07:17.550 --> 00:07:23.190 align:middle line:84% Now, we can invert that and ask for a single pixel 00:07:23.190 --> 00:07:27.600 align:middle line:84% of the camera, which projector pixels are contributing. 00:07:27.600 --> 00:07:30.120 align:middle line:84% So the question to ask is I'm going 00:07:30.120 --> 00:07:32.260 align:middle line:90% to turn on one pixel at a time. 00:07:32.260 --> 00:07:34.630 align:middle line:84% If I turn on just this array, for example, 00:07:34.630 --> 00:07:36.990 align:middle line:84% it will only contribute to, say, this pixel. 00:07:36.990 --> 00:07:38.730 align:middle line:84% But if there was some intersection, 00:07:38.730 --> 00:07:41.390 align:middle line:84% it will also contribute to some other pixel and so on. 00:07:41.390 --> 00:07:44.040 align:middle line:84% So you're going to have this global transport of light. 00:07:44.040 --> 00:07:47.970 align:middle line:84% And from that you can ask this question, for a given camera 00:07:47.970 --> 00:07:50.160 align:middle line:84% pixel, what are the other projector pixels that 00:07:50.160 --> 00:07:52.360 align:middle line:90% are contributing to it? 00:07:52.360 --> 00:07:54.850 align:middle line:84% And so you can do this inversion, 00:07:54.850 --> 00:08:00.270 align:middle line:84% which is relatively straightforward to do. 00:08:00.270 --> 00:08:02.670 align:middle line:84% And then you can do experiments like this, 00:08:02.670 --> 00:08:07.170 align:middle line:84% how can you read your opponent's cards from across the table? 00:08:07.170 --> 00:08:10.600 align:middle line:84% You can turn on the projector one pixel at a time, 00:08:10.600 --> 00:08:13.380 align:middle line:84% look at the reflection, and it's in the camera, 00:08:13.380 --> 00:08:16.990 align:middle line:84% then on the next pixel, and so on. 00:08:16.990 --> 00:08:19.380 align:middle line:84% And after you allow your opponent 00:08:19.380 --> 00:08:22.860 align:middle line:84% to take a million photos of him, you'll 00:08:22.860 --> 00:08:25.290 align:middle line:84% be able to read the card although it 00:08:25.290 --> 00:08:29.800 align:middle line:84% [? will ?] not directly visible from your point of view. 00:08:29.800 --> 00:08:33.299 align:middle line:84% So you can kind of look around an uploader 00:08:33.299 --> 00:08:35.700 align:middle line:90% and what's behind a corner. 00:08:35.700 --> 00:08:37.659 align:middle line:84% What's the-- what's the flaw in this argument 00:08:37.659 --> 00:08:38.909 align:middle line:90% that you can look in a corner? 00:08:38.909 --> 00:08:41.980 align:middle line:90% 00:08:41.980 --> 00:08:42.480 align:middle line:90% Yes. 00:08:42.480 --> 00:08:43.592 align:middle line:90% AUDIENCE: [INAUDIBLE]. 00:08:43.592 --> 00:08:44.550 align:middle line:90% RAMESH RASKAR: Exactly. 00:08:44.550 --> 00:08:45.925 align:middle line:84% There's some device that you have 00:08:45.925 --> 00:08:50.310 align:middle line:84% to place that's actually looking at that card. 00:08:50.310 --> 00:08:53.880 align:middle line:84% So this is a beautiful project from Stanford. 00:08:53.880 --> 00:08:57.142 align:middle line:84% And one of their funding agencies, of course, the army-- 00:08:57.142 --> 00:08:59.350 align:middle line:84% and they would like to know what's around the corner. 00:08:59.350 --> 00:09:00.933 align:middle line:84% But of course you have to go and place 00:09:00.933 --> 00:09:04.280 align:middle line:84% a projector in the enemy line to be able to figure out 00:09:04.280 --> 00:09:06.460 align:middle line:90% who's out there. 00:09:06.460 --> 00:09:10.890 align:middle line:84% So this is the message of this class. 00:09:10.890 --> 00:09:14.068 align:middle line:84% Pure digital cameras are extremely boring. 00:09:14.068 --> 00:09:16.110 align:middle line:84% And if you look at these two cameras, one of them 00:09:16.110 --> 00:09:16.650 align:middle line:90% is digital. 00:09:16.650 --> 00:09:18.390 align:middle line:90% One of them is phone. 00:09:18.390 --> 00:09:21.630 align:middle line:84% Can you really figure out which one is which? 00:09:21.630 --> 00:09:24.030 align:middle line:90% Left is digital. 00:09:24.030 --> 00:09:26.040 align:middle line:90% Right is digital. 00:09:26.040 --> 00:09:28.830 align:middle line:90% OK, and the rest are confused. 00:09:28.830 --> 00:09:31.590 align:middle line:84% So hardly any difference between them, 00:09:31.590 --> 00:09:35.780 align:middle line:84% zoom, focus, aperture exposure, all the same old boring stuff. 00:09:35.780 --> 00:09:37.440 align:middle line:90% There's not much extra going on. 00:09:37.440 --> 00:09:40.680 align:middle line:84% And if somebody claims that digital camera have spawned 00:09:40.680 --> 00:09:42.092 align:middle line:90% a new art form, probably not. 00:09:42.092 --> 00:09:43.800 align:middle line:84% I mean, you could just take film cameras, 00:09:43.800 --> 00:09:46.800 align:middle line:90% scan them, and play with that. 00:09:46.800 --> 00:09:50.820 align:middle line:84% They're just faster, better, and cheaper. 00:09:50.820 --> 00:09:56.160 align:middle line:84% So when people think about computational photography 00:09:56.160 --> 00:09:58.950 align:middle line:84% and computational camera, they start thinking about, 00:09:58.950 --> 00:10:01.680 align:middle line:84% wow, OK I have this camera that doesn't 00:10:01.680 --> 00:10:06.550 align:middle line:84% do a very good job of dynamic range and field of view 00:10:06.550 --> 00:10:07.790 align:middle line:90% and so on. 00:10:07.790 --> 00:10:09.540 align:middle line:84% So what I'm going to do is somehow improve 00:10:09.540 --> 00:10:12.000 align:middle line:90% the performance of this camera. 00:10:12.000 --> 00:10:14.910 align:middle line:84% One common is by boosting the dynamic range 00:10:14.910 --> 00:10:19.110 align:middle line:84% by taking multiple photo by exposure bracketing 00:10:19.110 --> 00:10:21.420 align:middle line:84% or having a larger depth of field 00:10:21.420 --> 00:10:25.320 align:middle line:84% by taking multiple photo by taking focus bracketing and so 00:10:25.320 --> 00:10:25.920 align:middle line:90% on. 00:10:25.920 --> 00:10:27.600 align:middle line:84% So to increase the field of view, 00:10:27.600 --> 00:10:31.360 align:middle line:84% you're going to do a panorama and then stitch it together. 00:10:31.360 --> 00:10:33.090 align:middle line:84% If you want to increase the frame rate, 00:10:33.090 --> 00:10:36.430 align:middle line:84% you're going to play with exposure time and so on. 00:10:36.430 --> 00:10:39.510 align:middle line:84% So this is what a lot of people think about as computation 00:10:39.510 --> 00:10:41.130 align:middle line:90% camera and photography. 00:10:41.130 --> 00:10:43.260 align:middle line:84% I just want to emphasize that this is not 00:10:43.260 --> 00:10:45.660 align:middle line:84% what we're going to talk about in this class. 00:10:45.660 --> 00:10:49.980 align:middle line:84% These are concepts literally easy to understand 00:10:49.980 --> 00:10:53.280 align:middle line:84% and I have a lot of lecture, notes, and videos online. 00:10:53.280 --> 00:10:54.480 align:middle line:90% You can look it up. 00:10:54.480 --> 00:10:55.500 align:middle line:90% In a couple of hours. 00:10:55.500 --> 00:10:58.050 align:middle line:84% You will get a pretty good overview of all the things 00:10:58.050 --> 00:11:00.130 align:middle line:90% that can be done in this space. 00:11:00.130 --> 00:11:03.450 align:middle line:84% And if you go to Flickr, there are 00:11:03.450 --> 00:11:05.700 align:middle line:84% groups that are just exploring high dynamic range 00:11:05.700 --> 00:11:06.780 align:middle line:90% imaging, et cetera. 00:11:06.780 --> 00:11:08.910 align:middle line:84% So we are not really going to talk about that 00:11:08.910 --> 00:11:10.710 align:middle line:84% because all these techniques are just 00:11:10.710 --> 00:11:13.740 align:middle line:84% trying to improve the performance of a camera. 00:11:13.740 --> 00:11:18.530 align:middle line:84% It's not trying to change the game of photography. 00:11:18.530 --> 00:11:22.030 align:middle line:84% So generalizing it even further, what we're going to look at 00:11:22.030 --> 00:11:26.080 align:middle line:84% is cameras that are not just 2D sensors mimicking film. 00:11:26.080 --> 00:11:28.660 align:middle line:84% But they could be zero-dimensional sensors that 00:11:28.660 --> 00:11:31.090 align:middle line:84% are, as I said in barcode scanner, 00:11:31.090 --> 00:11:34.390 align:middle line:84% and we'll look at time of flight, motion detection. 00:11:34.390 --> 00:11:37.240 align:middle line:84% One these sensors that you see in flatbed scanners or fax 00:11:37.240 --> 00:11:41.260 align:middle line:84% machines or line scan cameras that are used in photo finish 00:11:41.260 --> 00:11:44.950 align:middle line:84% at sports events, 2 and 1/2 new sensors. 00:11:44.950 --> 00:11:50.440 align:middle line:84% 3D sensors are a different kind and very quickly into 4D and 6D 00:11:50.440 --> 00:11:55.750 align:middle line:84% devices that are exploiting deconvolution and tomography 00:11:55.750 --> 00:11:58.990 align:middle line:84% in medical imaging or scientific imaging and also displays 00:11:58.990 --> 00:12:02.850 align:middle line:84% that are four-dimensional and six-dimensional. 00:12:02.850 --> 00:12:06.870 align:middle line:84% So earlier we saw that you can look around the corner 00:12:06.870 --> 00:12:10.600 align:middle line:84% by placing a device in the line of sight, 00:12:10.600 --> 00:12:12.900 align:middle line:90% which is the projector. 00:12:12.900 --> 00:12:17.820 align:middle line:84% Well, that was done three years ago or four years ago, 2005. 00:12:17.820 --> 00:12:19.980 align:middle line:84% Do we have some new machinery now 00:12:19.980 --> 00:12:22.980 align:middle line:84% that will allow us to look around the corner? 00:12:22.980 --> 00:12:23.655 align:middle line:90% You're out here. 00:12:23.655 --> 00:12:26.770 align:middle line:90% 00:12:26.770 --> 00:12:29.226 align:middle line:90% The meetings are called. 00:12:29.226 --> 00:12:30.050 align:middle line:90% It's a good day. 00:12:30.050 --> 00:12:34.190 align:middle line:90% 00:12:34.190 --> 00:12:36.200 align:middle line:84% You're out here, and you want to see 00:12:36.200 --> 00:12:40.202 align:middle line:84% what's inside with a door that's partially open. 00:12:40.202 --> 00:12:40.910 align:middle line:90% Is this possible? 00:12:40.910 --> 00:12:44.620 align:middle line:90% 00:12:44.620 --> 00:12:52.190 align:middle line:84% Or imagine your cameras that are right now in mobile devices. 00:12:52.190 --> 00:12:56.080 align:middle line:84% You have this thing that's shrinking every day, 00:12:56.080 --> 00:12:58.540 align:middle line:84% getting cheaper but also shrinking and actually 00:12:58.540 --> 00:13:02.110 align:middle line:84% degrading its signal to noise ratio performance. 00:13:02.110 --> 00:13:07.690 align:middle line:84% But imagine if your whole LCD becomes photosensitive, 00:13:07.690 --> 00:13:10.390 align:middle line:84% so every pixel that's emitting on your LCD 00:13:10.390 --> 00:13:12.940 align:middle line:90% also becomes photosensitive-- 00:13:12.940 --> 00:13:16.270 align:middle line:84% and companies like Sharp and Planar are already doing that. 00:13:16.270 --> 00:13:20.350 align:middle line:84% So they have this LCD array where 00:13:20.350 --> 00:13:22.900 align:middle line:84% if you put a finger directly on top of the LCD, 00:13:22.900 --> 00:13:28.810 align:middle line:84% you can even look at the ridges on this finger. 00:13:28.810 --> 00:13:30.910 align:middle line:84% So there's some very beautiful opportunities. 00:13:30.910 --> 00:13:32.870 align:middle line:84% And I just saw Kimo and Ted here. 00:13:32.870 --> 00:13:34.840 align:middle line:84% They have also a very exciting project 00:13:34.840 --> 00:13:36.640 align:middle line:84% where they can use an ordinary camera 00:13:36.640 --> 00:13:42.070 align:middle line:84% to look at right down to the resolution of your ridges. 00:13:42.070 --> 00:13:47.420 align:middle line:84% Now the question is, if we redefine a camera 00:13:47.420 --> 00:13:49.790 align:middle line:84% not to be this perspective device that 00:13:49.790 --> 00:13:51.530 align:middle line:90% behaves like a pinhole camera-- 00:13:51.530 --> 00:13:54.425 align:middle line:84% but the whole screen is like sensing. 00:13:54.425 --> 00:13:57.020 align:middle line:90% 00:13:57.020 --> 00:14:01.700 align:middle line:84% There's a fusion of sensor and optics. 00:14:01.700 --> 00:14:05.510 align:middle line:84% Think about in the display domain a CRT. 00:14:05.510 --> 00:14:09.170 align:middle line:84% There is an emitter, the electron beam, and then 00:14:09.170 --> 00:14:12.020 align:middle line:84% the receiver, which is the phosphorescent screen. 00:14:12.020 --> 00:14:15.290 align:middle line:84% There's always a separation between what's emitting it 00:14:15.290 --> 00:14:17.070 align:middle line:90% and what's receiving it. 00:14:17.070 --> 00:14:21.320 align:middle line:84% And over time, of course, we have a LCDs where 00:14:21.320 --> 00:14:22.883 align:middle line:90% everything is fused together. 00:14:22.883 --> 00:14:24.800 align:middle line:84% And if you think about cameras, it's the same. 00:14:24.800 --> 00:14:27.080 align:middle line:84% You have a lens that's collecting, 00:14:27.080 --> 00:14:29.540 align:middle line:84% and then you have a sensor that's receiving. 00:14:29.540 --> 00:14:32.420 align:middle line:84% And we haven't thought that there's always 00:14:32.420 --> 00:14:34.940 align:middle line:84% has to be some distance sufficient distance. 00:14:34.940 --> 00:14:36.650 align:middle line:84% If you have a 35 millimeter camera, 00:14:36.650 --> 00:14:41.000 align:middle line:84% you ought to have roughly 35 millimeters between your lens 00:14:41.000 --> 00:14:42.650 align:middle line:90% and your sensor. 00:14:42.650 --> 00:14:45.320 align:middle line:84% And we can throw away all those constraints and all 00:14:45.320 --> 00:14:47.030 align:middle line:84% those assumptions, and we might be 00:14:47.030 --> 00:14:48.950 align:middle line:84% able to come up with devices where 00:14:48.950 --> 00:14:52.940 align:middle line:84% the lens and the optics and the sensor is all just one thing. 00:14:52.940 --> 00:14:57.090 align:middle line:84% And that would be kind of the LCD equivalent of a camera. 00:14:57.090 --> 00:15:00.480 align:middle line:84% So this is something we can think about. 00:15:00.480 --> 00:15:02.210 align:middle line:84% And these devices are coming mainly 00:15:02.210 --> 00:15:04.970 align:middle line:84% because they want to support touch sensing. 00:15:04.970 --> 00:15:08.250 align:middle line:84% But imagine the future if you have your mobile device, 00:15:08.250 --> 00:15:11.520 align:middle line:84% and you can just wave it and take a picture-- 00:15:11.520 --> 00:15:13.550 align:middle line:84% and because your sensor is so large, 00:15:13.550 --> 00:15:15.890 align:middle line:84% you'll be able to collect a lot of light. 00:15:15.890 --> 00:15:19.380 align:middle line:84% And right now this it's not even megapixel yet. 00:15:19.380 --> 00:15:21.590 align:middle line:84% But this will be a megapixel, 10 megapixel. 00:15:21.590 --> 00:15:25.800 align:middle line:84% And you'll be able to multiplex those pixels for color 00:15:25.800 --> 00:15:29.870 align:middle line:84% and infrared and all kinds of beautiful things, 00:15:29.870 --> 00:15:32.100 align:middle line:90% different speeds, and so on. 00:15:32.100 --> 00:15:34.790 align:middle line:84% So that's how we're going to think about in this class 00:15:34.790 --> 00:15:38.870 align:middle line:90% about cameras and photography. 00:15:38.870 --> 00:15:41.690 align:middle line:84% And the same with medical imaging. 00:15:41.690 --> 00:15:44.180 align:middle line:84% Right now medical imaging is thought as-- 00:15:44.180 --> 00:15:47.690 align:middle line:84% medical imaging today is very similar to photography 00:15:47.690 --> 00:15:49.670 align:middle line:90% in the 1930s. 00:15:49.670 --> 00:15:53.390 align:middle line:84% The guy who took the photo was more important than the people 00:15:53.390 --> 00:15:55.130 align:middle line:84% whose photo was being taken sometimes, 00:15:55.130 --> 00:15:57.470 align:middle line:84% because those people have to go to the sky 00:15:57.470 --> 00:15:59.520 align:middle line:90% and stand for some time. 00:15:59.520 --> 00:16:03.050 align:middle line:84% And he had this one specialized device called a camera. 00:16:03.050 --> 00:16:05.420 align:middle line:84% Everybody has to stand still for a few seconds. 00:16:05.420 --> 00:16:07.880 align:middle line:84% You take a picture, and then you go home. 00:16:07.880 --> 00:16:10.640 align:middle line:84% And after a couple of days, you get your photo back. 00:16:10.640 --> 00:16:14.240 align:middle line:84% And that's exactly what medical imaging is all about today. 00:16:14.240 --> 00:16:16.280 align:middle line:84% We have to go to a special location 00:16:16.280 --> 00:16:20.330 align:middle line:84% to get our CAT scan or ultrasound scan done. 00:16:20.330 --> 00:16:23.720 align:middle line:90% And our time is completely-- 00:16:23.720 --> 00:16:26.300 align:middle line:84% our time is considered not so important. 00:16:26.300 --> 00:16:27.740 align:middle line:84% The guy who's running the machine, 00:16:27.740 --> 00:16:29.198 align:middle line:84% he's very important because we have 00:16:29.198 --> 00:16:32.640 align:middle line:84% to make an appointment with this guy. 00:16:32.640 --> 00:16:36.200 align:middle line:90% So can we bring-- 00:16:36.200 --> 00:16:38.540 align:middle line:84% will medical imaging evolve to a stage 00:16:38.540 --> 00:16:41.030 align:middle line:84% where it's like photography today? 00:16:41.030 --> 00:16:45.080 align:middle line:84% Clearly, when we take a photo if somebody spills a coffee, 00:16:45.080 --> 00:16:46.020 align:middle line:90% we take a photo. 00:16:46.020 --> 00:16:48.710 align:middle line:84% It's a very casual way of thinking about photography. 00:16:48.710 --> 00:16:50.990 align:middle line:84% And then medical imaging evolved to a stage 00:16:50.990 --> 00:16:52.570 align:middle line:90% where we can do that. 00:16:52.570 --> 00:16:53.570 align:middle line:90% And it turns out we can. 00:16:53.570 --> 00:16:56.630 align:middle line:84% There are certain directions, certain computational methods 00:16:56.630 --> 00:17:00.680 align:middle line:84% that we will be able to develop that will get rid 00:17:00.680 --> 00:17:07.510 align:middle line:84% of older mindsets of how medical devices are being developed. 00:17:07.510 --> 00:17:10.750 align:middle line:84% So a very brief introduction of where 00:17:10.750 --> 00:17:14.720 align:middle line:84% I come from and our group here Camera Culture. 00:17:14.720 --> 00:17:16.480 align:middle line:84% So in the past I have worked a lot 00:17:16.480 --> 00:17:18.819 align:middle line:84% on [? conventional ?] illumination, different types 00:17:18.819 --> 00:17:22.450 align:middle line:84% of projectors, creating multi projector displays, 00:17:22.450 --> 00:17:26.589 align:middle line:84% creating virtual reality setups, augmented 00:17:26.589 --> 00:17:29.230 align:middle line:84% reality, a lot of work on pocket projectors 00:17:29.230 --> 00:17:32.530 align:middle line:84% and augmented reality with pocket projectors, interaction 00:17:32.530 --> 00:17:36.590 align:middle line:84% paradigms, work with RFIDs and so on. 00:17:36.590 --> 00:17:40.720 align:middle line:84% And in the last eight years or so, a lot of work in cameras 00:17:40.720 --> 00:17:45.610 align:middle line:84% playing with shutter aperture, light fields, illumination 00:17:45.610 --> 00:17:49.400 align:middle line:84% wavelength, and sensors and so on. 00:17:49.400 --> 00:17:54.190 align:middle line:84% And some of the questions we discuss in our group are this. 00:17:54.190 --> 00:17:58.210 align:middle line:84% And what will the camera look like in 10 or 20 years 00:17:58.210 --> 00:17:59.170 align:middle line:90% from now? 00:17:59.170 --> 00:18:03.160 align:middle line:84% I already gave you one kind of possible direction 00:18:03.160 --> 00:18:05.590 align:middle line:84% where it might be just a flat screen, 00:18:05.590 --> 00:18:07.310 align:middle line:84% or maybe it's just your credit card. 00:18:07.310 --> 00:18:12.220 align:middle line:84% But it could even be a retina display retinal sensor 00:18:12.220 --> 00:18:14.440 align:middle line:90% and digital sensors. 00:18:14.440 --> 00:18:16.720 align:middle line:84% How will the next billion cameras 00:18:16.720 --> 00:18:21.160 align:middle line:84% change our social fabric or social culture? 00:18:21.160 --> 00:18:22.642 align:middle line:90% If you think about the-- 00:18:22.642 --> 00:18:24.100 align:middle line:84% if you think about the internet, it 00:18:24.100 --> 00:18:27.940 align:middle line:84% has been transformed by the ability to search. 00:18:27.940 --> 00:18:32.740 align:middle line:84% That includes indexing and segmenting and sorting and all 00:18:32.740 --> 00:18:34.420 align:middle line:90% those beautiful problems. 00:18:34.420 --> 00:18:37.240 align:middle line:84% But image search even today remains an extremely 00:18:37.240 --> 00:18:39.560 align:middle line:90% challenging problem. 00:18:39.560 --> 00:18:46.420 align:middle line:84% So maybe we can change the game and modify our devices, 00:18:46.420 --> 00:18:48.580 align:middle line:84% modify our cameras, modify our displays, 00:18:48.580 --> 00:18:51.530 align:middle line:84% modify our storage devices and so on. 00:18:51.530 --> 00:18:55.960 align:middle line:84% So that image search can become simpler. 00:18:55.960 --> 00:18:56.530 align:middle line:90% OK. 00:18:56.530 --> 00:18:58.810 align:middle line:84% If they magically have a camera that's 00:18:58.810 --> 00:19:02.350 align:middle line:84% 100 megapixel, when I take your photo, 00:19:02.350 --> 00:19:05.380 align:middle line:84% I can Zoom in all the way down to your iris. 00:19:05.380 --> 00:19:07.660 align:middle line:84% And if I have an iris detector, then I 00:19:07.660 --> 00:19:08.950 align:middle line:90% have very easy identification. 00:19:08.950 --> 00:19:11.230 align:middle line:84% And all the photos I take of Dina, for example, 00:19:11.230 --> 00:19:16.120 align:middle line:84% they're all indexed with her iris code. 00:19:16.120 --> 00:19:18.910 align:middle line:84% So that's kind of a very [INAUDIBLE] level 00:19:18.910 --> 00:19:21.970 align:middle line:84% way of thinking about simplifying image search. 00:19:21.970 --> 00:19:23.920 align:middle line:84% But we'll look at a very interesting example 00:19:23.920 --> 00:19:30.160 align:middle line:84% that exploit thermal IR imaging, electric cameras and so on. 00:19:30.160 --> 00:19:33.190 align:middle line:84% What will happen when for this recording is being 00:19:33.190 --> 00:19:34.900 align:middle line:84% used not just by the big brother, 00:19:34.900 --> 00:19:38.020 align:middle line:84% but it's actually being used for some beneficial purposes 00:19:38.020 --> 00:19:39.550 align:middle line:90% in commercial settings? 00:19:39.550 --> 00:19:44.380 align:middle line:84% So imagine a scenario something like Google Earth live 00:19:44.380 --> 00:19:47.470 align:middle line:84% where right now you can go to Google Earth 00:19:47.470 --> 00:19:52.120 align:middle line:84% and fire up the browser, and you can go to any part of the world 00:19:52.120 --> 00:19:58.750 align:middle line:84% and see how it looks about six months ago or a year ago. 00:19:58.750 --> 00:20:01.060 align:middle line:84% But imagine if you can just fire up your browser 00:20:01.060 --> 00:20:05.718 align:middle line:84% and go to this location live as it's happening now. 00:20:05.718 --> 00:20:08.260 align:middle line:84% And of course, you can move your slider and go back and forth 00:20:08.260 --> 00:20:09.680 align:middle line:90% and back. 00:20:09.680 --> 00:20:13.870 align:middle line:84% So if you have a camera on every street pole, every bus, 00:20:13.870 --> 00:20:17.380 align:middle line:84% every taxi, every person and all those data 00:20:17.380 --> 00:20:21.250 align:middle line:84% all that data is being streamed on the network, 00:20:21.250 --> 00:20:25.060 align:middle line:84% so you can go to any part of the world and see it, 00:20:25.060 --> 00:20:26.980 align:middle line:90% how will that world look like? 00:20:26.980 --> 00:20:31.300 align:middle line:84% I'm sure many of you are definitely scared-- 00:20:31.300 --> 00:20:36.160 align:middle line:84% and given all the privacy and security notions of it. 00:20:36.160 --> 00:20:40.690 align:middle line:84% But this could become very similar to the safety 00:20:40.690 --> 00:20:44.180 align:middle line:84% we have a little bit in our financial system 00:20:44.180 --> 00:20:45.400 align:middle line:90% our financial transactions. 00:20:45.400 --> 00:20:48.790 align:middle line:84% When I use my card in a restaurant, a bunch of people 00:20:48.790 --> 00:20:51.625 align:middle line:84% actually see my credit card number and all the information. 00:20:51.625 --> 00:20:53.770 align:middle line:90% The waitress looks at the card. 00:20:53.770 --> 00:20:54.850 align:middle line:90% And the owner knows it. 00:20:54.850 --> 00:20:56.080 align:middle line:90% The bank knows it. 00:20:56.080 --> 00:20:57.567 align:middle line:84% The credit card company knows it. 00:20:57.567 --> 00:20:59.650 align:middle line:84% And the government, of course, knows it and so on. 00:20:59.650 --> 00:21:02.108 align:middle line:84% And a lot of people know about your financial transactions. 00:21:02.108 --> 00:21:05.350 align:middle line:84% But somehow you are completely comfortable sharing 00:21:05.350 --> 00:21:07.180 align:middle line:90% your financial data. 00:21:07.180 --> 00:21:10.840 align:middle line:84% And similarly can we create cameras and photography 00:21:10.840 --> 00:21:14.080 align:middle line:84% and imaging infrastructure so that if you walk down 00:21:14.080 --> 00:21:17.050 align:middle line:84% the street, and all these cameras are looking at you, 00:21:17.050 --> 00:21:19.270 align:middle line:84% anybody in the world could be looking at your live? 00:21:19.270 --> 00:21:21.460 align:middle line:84% You feel completely secure that it's 00:21:21.460 --> 00:21:24.620 align:middle line:84% being used only by the right people for the right reasons. 00:21:24.620 --> 00:21:26.170 align:middle line:84% And if you don't want them to see it, 00:21:26.170 --> 00:21:29.740 align:middle line:84% you have some switch on you that says, I'm completely invisible. 00:21:29.740 --> 00:21:32.830 align:middle line:84% And you should be able to walk down the street. 00:21:32.830 --> 00:21:35.460 align:middle line:84% Can we create such an imaging infrastructure? 00:21:35.460 --> 00:21:38.100 align:middle line:90% 00:21:38.100 --> 00:21:40.590 align:middle line:84% Think about high speed cameras and high resolution cameras. 00:21:40.590 --> 00:21:43.350 align:middle line:84% Maybe we'll have microscopes and nano scopes with us 00:21:43.350 --> 00:21:47.550 align:middle line:84% that will again change the way we think about medical imaging. 00:21:47.550 --> 00:21:50.700 align:middle line:84% And what about movie making and news reporting? 00:21:50.700 --> 00:21:54.540 align:middle line:84% As you know it has dramatically changed over the last three 00:21:54.540 --> 00:21:56.550 align:middle line:84% or four years because, again, a billion people 00:21:56.550 --> 00:21:58.200 align:middle line:90% have cameras out there. 00:21:58.200 --> 00:22:01.950 align:middle line:84% Whether there is an in Tibet or there's 00:22:01.950 --> 00:22:05.100 align:middle line:84% a satellite image of what's going on in Burma 00:22:05.100 --> 00:22:10.350 align:middle line:84% or a plane that's about to crash, you get amazing videos. 00:22:10.350 --> 00:22:12.150 align:middle line:90% And they're not captured by CNN. 00:22:12.150 --> 00:22:13.540 align:middle line:84% They're captured by other people. 00:22:13.540 --> 00:22:17.040 align:middle line:84% So how are we going to, again, change 00:22:17.040 --> 00:22:20.160 align:middle line:84% this imaging infrastructure, the whole pipeline, 00:22:20.160 --> 00:22:27.690 align:middle line:84% to think about the future of moviemaking and news reporting? 00:22:27.690 --> 00:22:30.120 align:middle line:84% So overall in this course, we're not 00:22:30.120 --> 00:22:32.650 align:middle line:84% going to think purely about software. 00:22:32.650 --> 00:22:37.200 align:middle line:84% We're going to think about how we can change the camera, 00:22:37.200 --> 00:22:38.430 align:middle line:90% not just use the camera. 00:22:38.430 --> 00:22:40.350 align:middle line:84% So that involves optics and illumination 00:22:40.350 --> 00:22:42.720 align:middle line:84% and sensor and motion of the sensor, 00:22:42.720 --> 00:22:48.030 align:middle line:84% motion of the optics, the different wavelengths, 3D 00:22:48.030 --> 00:22:53.280 align:middle line:84% cameras, polarization, probes and actuators and also priors 00:22:53.280 --> 00:22:58.620 align:middle line:84% and online collections, the network. 00:22:58.620 --> 00:23:03.030 align:middle line:84% And one kind of theme you'll see is 00:23:03.030 --> 00:23:06.270 align:middle line:84% after years of research in computer vision, one 00:23:06.270 --> 00:23:09.950 align:middle line:84% could argue that we have exhausted the bits that 00:23:09.950 --> 00:23:10.950 align:middle line:90% are available in pixels. 00:23:10.950 --> 00:23:12.160 align:middle line:90% There's still a lot to be done. 00:23:12.160 --> 00:23:13.202 align:middle line:90% There's a lot to be done. 00:23:13.202 --> 00:23:15.600 align:middle line:90% 00:23:15.600 --> 00:23:18.300 align:middle line:84% Still challenging even today with sophisticated computing 00:23:18.300 --> 00:23:19.930 align:middle line:90% algorithms, it's challenging. 00:23:19.930 --> 00:23:23.250 align:middle line:84% So maybe we can build feature revealing cameras 00:23:23.250 --> 00:23:29.730 align:middle line:84% that will go hand in hand with existing or modern [INAUDIBLE] 00:23:29.730 --> 00:23:33.750 align:middle line:84% algorithms so that we can process photons and create 00:23:33.750 --> 00:23:39.130 align:middle line:84% this meta structure for our imaging pipeline. 00:23:39.130 --> 00:23:42.270 align:middle line:84% So what we're going to do today is 00:23:42.270 --> 00:23:45.330 align:middle line:84% I'll briefly describe what this course is about, 00:23:45.330 --> 00:23:46.920 align:middle line:90% do some introductions. 00:23:46.920 --> 00:23:49.170 align:middle line:84% And the second half, we will come around 00:23:49.170 --> 00:23:55.990 align:middle line:84% and do a fast forward preview of the [? core ?] groups, OK. 00:23:55.990 --> 00:23:57.990 align:middle line:84% So in the next few slides, I'm just 00:23:57.990 --> 00:24:00.480 align:middle line:84% going to give you a quick kind of rundown 00:24:00.480 --> 00:24:03.090 align:middle line:84% of what this course is about, the layout. 00:24:03.090 --> 00:24:12.860 align:middle line:90% 00:24:12.860 --> 00:24:20.750 align:middle line:84% Here's a nice overview of the biological successful vision 00:24:20.750 --> 00:24:22.460 align:middle line:90% in the animal eyes. 00:24:22.460 --> 00:24:25.640 align:middle line:84% And there's a nice paper in Science. 00:24:25.640 --> 00:24:29.570 align:middle line:84% And you can take all the successful biological vision 00:24:29.570 --> 00:24:33.710 align:middle line:84% and place that in kind of eight categories. 00:24:33.710 --> 00:24:37.550 align:middle line:84% You have eyes based on shadows, based on refraction, 00:24:37.550 --> 00:24:41.570 align:middle line:84% based on reflection, single chambered eyes, 00:24:41.570 --> 00:24:45.350 align:middle line:84% or compound eyes sometimes with apposition, 00:24:45.350 --> 00:24:47.900 align:middle line:84% sometimes with superposition, and so on. 00:24:47.900 --> 00:24:49.700 align:middle line:84% We'll come back and discuss this more. 00:24:49.700 --> 00:24:54.920 align:middle line:84% If you look at the eyes of a scallop, 00:24:54.920 --> 00:24:57.200 align:middle line:84% it's based on a mirror, not on a lens. 00:24:57.200 --> 00:25:01.970 align:middle line:84% So you have a mirror down here, and the sensor is up here. 00:25:01.970 --> 00:25:06.260 align:middle line:84% And light actually reflects from this concave mirror. 00:25:06.260 --> 00:25:09.050 align:middle line:84% And the image is found on the detector. 00:25:09.050 --> 00:25:15.290 align:middle line:84% And the future camera in your mobile device on a flat sensor 00:25:15.290 --> 00:25:17.720 align:middle line:90% could be on this architecture. 00:25:17.720 --> 00:25:21.570 align:middle line:84% It doesn't have to be on this single chamber high refraction. 00:25:21.570 --> 00:25:25.340 align:middle line:84% So the human vision is in this corner here. 00:25:25.340 --> 00:25:28.835 align:middle line:84% It's a single chamber design with lenses. 00:25:28.835 --> 00:25:32.300 align:middle line:84% And all the cameras, at least all the standard cameras 00:25:32.300 --> 00:25:36.750 align:middle line:84% that we know of are in this particular part. 00:25:36.750 --> 00:25:39.050 align:middle line:90% But this is a lot to explore. 00:25:39.050 --> 00:25:40.910 align:middle line:84% And that doesn't even explore computation. 00:25:40.910 --> 00:25:45.710 align:middle line:84% This is pretty much raw imaging that these animals are using. 00:25:45.710 --> 00:25:50.000 align:middle line:84% So film light, traditional photography, light 00:25:50.000 --> 00:25:51.630 align:middle line:90% comes in through the lens. 00:25:51.630 --> 00:25:53.687 align:middle line:84% It falls on the detector of the film, 00:25:53.687 --> 00:25:55.020 align:middle line:90% and that's the end of the story. 00:25:55.020 --> 00:25:55.812 align:middle line:90% That's your photon. 00:25:55.812 --> 00:25:57.980 align:middle line:84% And you just transfer it digitally 00:25:57.980 --> 00:26:02.750 align:middle line:84% or by chemical processing, and you should see it. 00:26:02.750 --> 00:26:05.550 align:middle line:84% Computational camera is a little bit different. 00:26:05.550 --> 00:26:08.840 align:middle line:84% We're going to have some crazy optics that's 00:26:08.840 --> 00:26:11.150 align:middle line:84% going to think about how rays and wavelengths are 00:26:11.150 --> 00:26:12.440 align:middle line:90% manipulated. 00:26:12.440 --> 00:26:15.680 align:middle line:84% We're going to have a sensor that's not just mimicking film. 00:26:15.680 --> 00:26:17.510 align:middle line:84% It's not going to be a [? flag ?] sensor. 00:26:17.510 --> 00:26:22.640 align:middle line:84% It'll have its own geometry and spectrum and so on. 00:26:22.640 --> 00:26:27.110 align:middle line:84% And what you see finally will not be just a raw image 00:26:27.110 --> 00:26:30.650 align:middle line:84% but will have some reconstruction and computation 00:26:30.650 --> 00:26:31.190 align:middle line:90% on top. 00:26:31.190 --> 00:26:32.482 align:middle line:90% So that's computational camera. 00:26:32.482 --> 00:26:37.410 align:middle line:90% 00:26:37.410 --> 00:26:39.623 align:middle line:84% That would be what's in the conventional camera. 00:26:39.623 --> 00:26:41.040 align:middle line:84% But there's also an element that's 00:26:41.040 --> 00:26:42.930 align:middle line:84% outside the camera, which is like-- 00:26:42.930 --> 00:26:45.330 align:middle line:84% and it's sort of just a flashlight. 00:26:45.330 --> 00:26:47.220 align:middle line:84% You're going to have a flashlight 00:26:47.220 --> 00:26:50.970 align:middle line:84% with sophisticated modulators that are changing the intensity 00:26:50.970 --> 00:26:55.810 align:middle line:84% and phase and polarization of light in different directions, 00:26:55.810 --> 00:26:58.810 align:middle line:84% different kinds of additional optics, and so on. 00:26:58.810 --> 00:27:00.750 align:middle line:84% So once you have a programmable light 00:27:00.750 --> 00:27:03.810 align:middle line:84% and a computational camera, we have a framework 00:27:03.810 --> 00:27:08.430 align:middle line:84% to really exploit and understand the light transport 00:27:08.430 --> 00:27:11.170 align:middle line:90% in the scene. 00:27:11.170 --> 00:27:16.330 align:middle line:84% So something interesting is happening in the camera work. 00:27:16.330 --> 00:27:19.350 align:middle line:90% We are here 2008, 2009. 00:27:19.350 --> 00:27:21.840 align:middle line:84% About a billion cameras are being 00:27:21.840 --> 00:27:26.550 align:middle line:90% sold every year, which is fine. 00:27:26.550 --> 00:27:27.600 align:middle line:90% Most people know that. 00:27:27.600 --> 00:27:30.560 align:middle line:84% But if you just go back about six years, 00:27:30.560 --> 00:27:38.180 align:middle line:84% 2001, 2002, 0 cameras were sold embedded in a mobile phone. 00:27:38.180 --> 00:27:45.240 align:middle line:84% So we have gone from 0 to a billion in just six years. 00:27:45.240 --> 00:27:51.120 align:middle line:84% So it's just an amazing time to think about imaging. 00:27:51.120 --> 00:27:54.110 align:middle line:84% And this is very much like all the fun 00:27:54.110 --> 00:27:55.820 align:middle line:84% in networking and communication back 00:27:55.820 --> 00:27:59.570 align:middle line:84% in the '90s when millions of people were coming on board 00:27:59.570 --> 00:28:02.960 align:middle line:84% and thousands of websites were going on. 00:28:02.960 --> 00:28:06.530 align:middle line:90% And that was computing. 00:28:06.530 --> 00:28:09.500 align:middle line:84% And this is the time for visual computing. 00:28:09.500 --> 00:28:12.140 align:middle line:84% And because this game is changing so fast 00:28:12.140 --> 00:28:16.460 align:middle line:84% in terms of cost, in terms of performance, 00:28:16.460 --> 00:28:21.461 align:middle line:84% applications where imaging was considered 00:28:21.461 --> 00:28:24.950 align:middle line:84% not the perfect solution is changing. 00:28:24.950 --> 00:28:29.300 align:middle line:84% We are seeing cameras being used in some really casual and very 00:28:29.300 --> 00:28:31.970 align:middle line:90% strange ways. 00:28:31.970 --> 00:28:34.640 align:middle line:84% And here's an indication of where that's going. 00:28:34.640 --> 00:28:36.410 align:middle line:90% So where are these cameras? 00:28:36.410 --> 00:28:40.850 align:middle line:84% Remember, cameras are not just 2D sensors for photography, 00:28:40.850 --> 00:28:44.940 align:middle line:84% but they're used in various ways. 00:28:44.940 --> 00:28:49.340 align:middle line:90% So we are here, 2008, 2009. 00:28:49.340 --> 00:28:52.880 align:middle line:84% The pink here is because of mobile phones. 00:28:52.880 --> 00:28:56.780 align:middle line:84% About a billion sensors are being sold. 00:28:56.780 --> 00:28:59.570 align:middle line:84% Any guesses about what these other slivers 00:28:59.570 --> 00:29:03.680 align:middle line:84% are, the blue one, the green one, and so on? 00:29:03.680 --> 00:29:08.150 align:middle line:84% There are a couple of million here. 00:29:08.150 --> 00:29:12.032 align:middle line:84% About 100 million here are being sold. 00:29:12.032 --> 00:29:14.170 align:middle line:90% I guess [INAUDIBLE]. 00:29:14.170 --> 00:29:14.670 align:middle line:90% Sorry. 00:29:14.670 --> 00:29:15.990 align:middle line:90% AUDIENCE: Optical mouse. 00:29:15.990 --> 00:29:18.816 align:middle line:84% RAMESH RASKAR: Optical mouse, very good, very good. 00:29:18.816 --> 00:29:19.603 align:middle line:90% AUDIENCE: Gaming. 00:29:19.603 --> 00:29:20.520 align:middle line:90% RAMESH RASKAR: Gaming. 00:29:20.520 --> 00:29:21.960 align:middle line:90% So if you think about-- 00:29:21.960 --> 00:29:25.800 align:middle line:84% we, I think, sold like 30 million remotes. 00:29:25.800 --> 00:29:27.307 align:middle line:90% 30 million remotes were sold. 00:29:27.307 --> 00:29:29.140 align:middle line:84% And this chart is actually made before that. 00:29:29.140 --> 00:29:32.880 align:middle line:84% So that's not even here, but that's going to be big, yes. 00:29:32.880 --> 00:29:36.090 align:middle line:84% And of course, there is traditional photography. 00:29:36.090 --> 00:29:39.300 align:middle line:84% But I'm glad you got optical mouse because that's 00:29:39.300 --> 00:29:40.710 align:middle line:90% one of the largest markets. 00:29:40.710 --> 00:29:43.110 align:middle line:84% It's basically a very low resolution, 00:29:43.110 --> 00:29:46.530 align:middle line:84% usually 20x20 or 32x32 pixel camera that's 00:29:46.530 --> 00:29:48.780 align:middle line:90% running 1,000 or 2,000 Hz. 00:29:48.780 --> 00:29:51.120 align:middle line:84% It's a high speed camera that's doing optical flow 00:29:51.120 --> 00:29:53.640 align:middle line:84% to figure out where your mouse is even if you put it 00:29:53.640 --> 00:29:57.810 align:middle line:90% on a very clear surface. 00:29:57.810 --> 00:30:00.240 align:middle line:84% And the mobile phones and digital cameras-- 00:30:00.240 --> 00:30:04.920 align:middle line:84% and all this worry about big brother, think about security. 00:30:04.920 --> 00:30:08.680 align:middle line:90% It's a very, very tiny sliver. 00:30:08.680 --> 00:30:12.570 align:middle line:84% And if you think about the first three categories here, 00:30:12.570 --> 00:30:17.970 align:middle line:84% optical mouse mobile phones, and digital and video cameras, 00:30:17.970 --> 00:30:19.620 align:middle line:90% these are all personal devices. 00:30:19.620 --> 00:30:21.990 align:middle line:84% And this is going to scale with the number of people 00:30:21.990 --> 00:30:23.070 align:middle line:90% in the world. 00:30:23.070 --> 00:30:26.190 align:middle line:84% So this will easily become 6 billion or 6.5 billion, 00:30:26.190 --> 00:30:27.780 align:middle line:90% whatever we have. 00:30:27.780 --> 00:30:30.090 align:middle line:84% But these other slivers may or may not grow. 00:30:30.090 --> 00:30:32.730 align:middle line:84% Gaming, for example, could still grow very rapidly 00:30:32.730 --> 00:30:35.220 align:middle line:84% because almost every person individually 00:30:35.220 --> 00:30:38.230 align:middle line:90% might own a game console. 00:30:38.230 --> 00:30:41.910 align:middle line:90% So very interesting here. 00:30:41.910 --> 00:30:46.770 align:middle line:84% If we look around the internet, cameras are being-- 00:30:46.770 --> 00:30:49.530 align:middle line:90% sometimes it's just silly. 00:30:49.530 --> 00:30:52.200 align:middle line:84% You may know about this do it yourself 00:30:52.200 --> 00:30:55.140 align:middle line:84% green screen effects company, U-Start. 00:30:55.140 --> 00:30:59.260 align:middle line:84% And they want to be the Guitar Hero in the visual domain. 00:30:59.260 --> 00:31:01.320 align:middle line:84% So instead of playing music that's 00:31:01.320 --> 00:31:04.830 align:middle line:84% synchronized with some prerecorded data, here 00:31:04.830 --> 00:31:06.750 align:middle line:84% they have pre-recorded video segments. 00:31:06.750 --> 00:31:08.730 align:middle line:84% And then you can star in this movie. 00:31:08.730 --> 00:31:10.830 align:middle line:90% It's pretty big. 00:31:10.830 --> 00:31:13.330 align:middle line:84% If you go to their website, a lot of people 00:31:13.330 --> 00:31:16.293 align:middle line:84% are uploading their videos with their software 00:31:16.293 --> 00:31:17.460 align:middle line:90% and their experience screen. 00:31:17.460 --> 00:31:19.690 align:middle line:84% All they give you is a camera, a simple camera, 00:31:19.690 --> 00:31:21.990 align:middle line:84% not even a studio pair, just a simple camera 00:31:21.990 --> 00:31:24.150 align:middle line:90% and a big plastic green screen. 00:31:24.150 --> 00:31:28.990 align:middle line:84% And they have some software for doing screen mapping and so on. 00:31:28.990 --> 00:31:33.283 align:middle line:84% You may be familiar with this video. 00:31:33.283 --> 00:31:34.200 align:middle line:90% Which company is that? 00:31:34.200 --> 00:31:35.070 align:middle line:90% I forgot. 00:31:35.070 --> 00:31:38.430 align:middle line:84% It's starts with M. It's a studio webcam. 00:31:38.430 --> 00:31:39.690 align:middle line:90% AUDIENCE: [INAUDIBLE]. 00:31:39.690 --> 00:31:42.300 align:middle line:84% RAMESH RASKAR: It's a Japanese company, OK. 00:31:42.300 --> 00:31:44.580 align:middle line:84% Some other interesting ones, look at this one here. 00:31:44.580 --> 00:31:47.340 align:middle line:84% Panasonic Beauty Appliance, what does it do? 00:31:47.340 --> 00:31:47.925 align:middle line:90% Any guesses? 00:31:47.925 --> 00:31:53.850 align:middle line:90% 00:31:53.850 --> 00:31:56.130 align:middle line:90% It's a pretty big camera. 00:31:56.130 --> 00:31:59.880 align:middle line:84% It's a personal beauty appliance where 00:31:59.880 --> 00:32:03.540 align:middle line:84% it provides humidification in a very local spot. 00:32:03.540 --> 00:32:07.590 align:middle line:84% So you can sit there and sit all day around and it I guess 00:32:07.590 --> 00:32:11.690 align:middle line:84% maintains the right level of moisture and humidity around. 00:32:11.690 --> 00:32:13.747 align:middle line:90% So your skin won't get very dry. 00:32:13.747 --> 00:32:14.580 align:middle line:90% What about this one? 00:32:14.580 --> 00:32:17.400 align:middle line:84% The camera is not anywhere near the eyesight. 00:32:17.400 --> 00:32:20.477 align:middle line:84% But the camera is actually on the ear. 00:32:20.477 --> 00:32:21.435 align:middle line:90% Can I look inside this? 00:32:21.435 --> 00:32:25.440 align:middle line:90% 00:32:25.440 --> 00:32:26.860 align:middle line:90% I was reading up on this. 00:32:26.860 --> 00:32:28.830 align:middle line:84% It wasn't very clear what the applications are. 00:32:28.830 --> 00:32:30.150 align:middle line:90% But you can think of it. 00:32:30.150 --> 00:32:32.040 align:middle line:84% I mean, if you're walking down the street 00:32:32.040 --> 00:32:34.995 align:middle line:84% and you want to be safe, something's coming at you. 00:32:34.995 --> 00:32:35.970 align:middle line:90% Maybe it tells you. 00:32:35.970 --> 00:32:38.490 align:middle line:84% Or if there's somebody you don't like-- 00:32:38.490 --> 00:32:40.260 align:middle line:90% they're walking towards you-- 00:32:40.260 --> 00:32:43.275 align:middle line:90% don't turn around. 00:32:43.275 --> 00:32:43.950 align:middle line:90% Yeah. 00:32:43.950 --> 00:32:48.270 align:middle line:84% And that's going to happen more and more, more crazy places. 00:32:48.270 --> 00:32:51.270 align:middle line:84% I like this GigaPan Epic 100 which is 00:32:51.270 --> 00:32:53.040 align:middle line:90% for creating panoramic imagery. 00:32:53.040 --> 00:32:56.730 align:middle line:84% But it's actually basically a tiny, tiny robotic platform 00:32:56.730 --> 00:32:59.730 align:middle line:90% about $300 for an [INAUDIBLE]. 00:32:59.730 --> 00:33:05.910 align:middle line:84% And it has a physical lever that presses on the shutter as it 00:33:05.910 --> 00:33:07.020 align:middle line:90% rotates around. 00:33:07.020 --> 00:33:09.060 align:middle line:90% It's pretty amazing. 00:33:09.060 --> 00:33:13.170 align:middle line:84% And of course Fuji is doing some fantastic work just 00:33:13.170 --> 00:33:14.505 align:middle line:90% starting with the studio pair. 00:33:14.505 --> 00:33:17.130 align:middle line:84% And they're trying to provide a whole pipeline all the way down 00:33:17.130 --> 00:33:19.440 align:middle line:84% to printing on a lenticular screen 00:33:19.440 --> 00:33:23.730 align:middle line:84% so you can enjoy this studio box. 00:33:23.730 --> 00:33:26.977 align:middle line:84% OK, so after you look at all this camera, 00:33:26.977 --> 00:33:28.560 align:middle line:84% they say, OK, what else is there to do 00:33:28.560 --> 00:33:31.680 align:middle line:84% because you don't have to do much research to do 00:33:31.680 --> 00:33:32.940 align:middle line:90% any one of these projects. 00:33:32.940 --> 00:33:38.800 align:middle line:84% You don't have to take a class to build these applications. 00:33:38.800 --> 00:33:40.870 align:middle line:84% So let's think about something trying to improve 00:33:40.870 --> 00:33:42.720 align:middle line:90% the camera just a little bit. 00:33:42.720 --> 00:33:47.160 align:middle line:84% All I want is depth per pixel, something humans 00:33:47.160 --> 00:33:49.152 align:middle line:90% seem to be doing very well. 00:33:49.152 --> 00:33:53.280 align:middle line:84% We can use our two eyes and figure out how far things are. 00:33:53.280 --> 00:33:55.692 align:middle line:84% At least we think we know how things are. 00:33:55.692 --> 00:33:57.150 align:middle line:84% There's a lot of prior information. 00:33:57.150 --> 00:34:02.580 align:middle line:84% But for a camera just to get depth is extremely challenging. 00:34:02.580 --> 00:34:05.940 align:middle line:84% After 34 years of research in computer vision, 00:34:05.940 --> 00:34:08.159 align:middle line:84% still you can't go out there and buy 00:34:08.159 --> 00:34:11.550 align:middle line:84% a camera that's a reasonable cost, which 00:34:11.550 --> 00:34:13.620 align:middle line:84% runs at a reasonable speed and gives you 00:34:13.620 --> 00:34:16.520 align:middle line:90% that perfect [INAUDIBLE]. 00:34:16.520 --> 00:34:17.870 align:middle line:90% It's amazing. 00:34:17.870 --> 00:34:20.000 align:middle line:84% So we'll show you what the state of the art 00:34:20.000 --> 00:34:25.269 align:middle line:84% is using this multi-thousand dollar camera. 00:34:25.269 --> 00:34:30.380 align:middle line:84% This was donated to us by [INAUDIBLE].. 00:34:30.380 --> 00:34:33.376 align:middle line:90% And [INAUDIBLE]. 00:34:33.376 --> 00:34:36.840 align:middle line:90% 00:34:36.840 --> 00:34:47.530 align:middle line:84% So this is camera that uses time of flight to [INAUDIBLE].. 00:34:47.530 --> 00:34:51.080 align:middle line:84% Oh, we don't use [INAUDIBLE] cameras as much. 00:34:51.080 --> 00:34:54.260 align:middle line:90% OK, so this is time of flight. 00:34:54.260 --> 00:34:56.210 align:middle line:90% All those alleles are being-- 00:34:56.210 --> 00:34:58.760 align:middle line:84% those of you who have a cell phone camera, just 00:34:58.760 --> 00:35:02.670 align:middle line:84% look at this lights through your camera. 00:35:02.670 --> 00:35:07.980 align:middle line:84% You'll see all of them are lit up in the air. 00:35:07.980 --> 00:35:11.174 align:middle line:90% And as you can see here-- 00:35:11.174 --> 00:35:12.430 align:middle line:90% what is it looking at? 00:35:12.430 --> 00:35:13.602 align:middle line:90% Is it [INAUDIBLE]? 00:35:13.602 --> 00:35:16.260 align:middle line:90% 00:35:16.260 --> 00:35:18.180 align:middle line:84% OK, so the people in the front are 00:35:18.180 --> 00:35:21.480 align:middle line:84% marked in green and somebody in the back over there. 00:35:21.480 --> 00:35:22.950 align:middle line:90% Yeah, Martin. 00:35:22.950 --> 00:35:28.020 align:middle line:84% So it's giving you some estimate of depth by doing 00:35:28.020 --> 00:35:29.610 align:middle line:90% a time of light calculation. 00:35:29.610 --> 00:35:32.110 align:middle line:90% Now that is the camera. 00:35:32.110 --> 00:35:33.120 align:middle line:90% And it works OK. 00:35:33.120 --> 00:35:35.490 align:middle line:84% But as you can see the quality even 00:35:35.490 --> 00:35:38.790 align:middle line:84% in this relatively easy configuration, 00:35:38.790 --> 00:35:40.830 align:middle line:84% most of the objects are diffused. 00:35:40.830 --> 00:35:42.090 align:middle line:90% There is no resistance. 00:35:42.090 --> 00:35:44.010 align:middle line:84% The movement doesn't have some light 00:35:44.010 --> 00:35:46.110 align:middle line:90% to overwhelm this active light. 00:35:46.110 --> 00:35:48.660 align:middle line:90% And even then the quality is OK. 00:35:48.660 --> 00:35:51.520 align:middle line:90% It's not that great. 00:35:51.520 --> 00:35:53.670 align:middle line:84% And I forget the exact cost of this one, 00:35:53.670 --> 00:35:58.560 align:middle line:84% but they cost about $10,000, these cameras. 00:35:58.560 --> 00:35:59.060 align:middle line:90% Go ahead. 00:35:59.060 --> 00:36:00.170 align:middle line:90% AUDIENCE: What's is it called? 00:36:00.170 --> 00:36:01.700 align:middle line:84% RAMESH RASKAR: This one is Canesta. 00:36:01.700 --> 00:36:05.690 align:middle line:84% Yeah, but during the later classes 00:36:05.690 --> 00:36:08.240 align:middle line:84% we'll learn about all the different 3D 00:36:08.240 --> 00:36:11.780 align:middle line:84% cameras, whether they're based on active light or stereo 00:36:11.780 --> 00:36:15.580 align:middle line:84% or structured light, polarization, and so on. 00:36:15.580 --> 00:36:16.830 align:middle line:90% So this one is going to start. 00:36:16.830 --> 00:36:24.160 align:middle line:84% The other company is 3DV, which was recently 00:36:24.160 --> 00:36:25.270 align:middle line:90% bought by Microsoft. 00:36:25.270 --> 00:36:27.820 align:middle line:90% 00:36:27.820 --> 00:36:32.590 align:middle line:84% And then there's a couple of companies in Germany 00:36:32.590 --> 00:36:35.350 align:middle line:90% who are building 3D cameras. 00:36:35.350 --> 00:36:37.570 align:middle line:84% And apparently this market is now 00:36:37.570 --> 00:36:40.870 align:middle line:84% being driven by game consoles because PlayStation 00:36:40.870 --> 00:36:43.840 align:middle line:84% and Xbox are all interested in using 00:36:43.840 --> 00:36:48.770 align:middle line:84% 3D cameras in the house for gaming, 00:36:48.770 --> 00:36:50.210 align:middle line:90% for interacting with gestures. 00:36:50.210 --> 00:36:53.530 align:middle line:84% So they want to distinguish this from this, for example, 00:36:53.530 --> 00:36:56.350 align:middle line:84% because right now all the games are in the gulag. 00:36:56.350 --> 00:36:59.560 align:middle line:84% The hands are away from your body to play the games. 00:36:59.560 --> 00:37:01.880 align:middle line:90% I-toy and Xbox and so on. 00:37:01.880 --> 00:37:05.620 align:middle line:84% But the new ones will allow you to do gestures 00:37:05.620 --> 00:37:09.680 align:middle line:90% that are more intricate. 00:37:09.680 --> 00:37:13.270 align:middle line:84% So there are too many challenges just 00:37:13.270 --> 00:37:17.530 align:middle line:84% to get 3D because we have to use some kind of active light 00:37:17.530 --> 00:37:21.250 align:middle line:84% to compensate for ambient scenes. 00:37:21.250 --> 00:37:27.130 align:middle line:84% You have to stitch geometric capture from multiple views. 00:37:27.130 --> 00:37:30.950 align:middle line:84% If you have something like marble or skin, 00:37:30.950 --> 00:37:32.260 align:middle line:90% it has subsurface scattering. 00:37:32.260 --> 00:37:33.910 align:middle line:84% So that's difficult to deal with. 00:37:33.910 --> 00:37:36.280 align:middle line:90% I cannot do triangulation. 00:37:36.280 --> 00:37:41.260 align:middle line:84% And objects that are diffused have diffuse reflection 00:37:41.260 --> 00:37:41.950 align:middle line:90% component. 00:37:41.950 --> 00:37:45.640 align:middle line:84% If you go into something that has glass or are dark, 00:37:45.640 --> 00:37:47.980 align:middle line:90% it's just out of caution. 00:37:47.980 --> 00:37:52.840 align:middle line:84% So computational camera, it should do something 00:37:52.840 --> 00:37:56.290 align:middle line:84% more than just capturing a 2D image. 00:37:56.290 --> 00:38:00.400 align:middle line:84% And those of you who are here building real time and C 00:38:00.400 --> 00:38:06.130 align:middle line:84% applications for robotics, anything like that. 00:38:06.130 --> 00:38:08.980 align:middle line:84% There's no camera right now that can 00:38:08.980 --> 00:38:10.240 align:middle line:90% deal with objects like this. 00:38:10.240 --> 00:38:11.950 align:middle line:90% This is amazing. 00:38:11.950 --> 00:38:14.210 align:middle line:84% So we have a billion cameras out there. 00:38:14.210 --> 00:38:17.270 align:middle line:84% But none of those cameras can solve this problem. 00:38:17.270 --> 00:38:19.440 align:middle line:84% So that's what I want to look at in this class. 00:38:19.440 --> 00:38:21.440 align:middle line:84% AUDIENCE: Can I ask a quick question about that? 00:38:21.440 --> 00:38:21.700 align:middle line:90% RAMESH RASKAR: Yeah. 00:38:21.700 --> 00:38:23.230 align:middle line:84% AUDIENCE: So I'm just thinking back 00:38:23.230 --> 00:38:26.260 align:middle line:84% to the traditional film type camera that 00:38:26.260 --> 00:38:29.350 align:middle line:84% has autofocus sensors that are contrast based 00:38:29.350 --> 00:38:30.970 align:middle line:90% or a rangefinder type thing. 00:38:30.970 --> 00:38:36.550 align:middle line:84% So that in a way gives you depth by moving optical elements. 00:38:36.550 --> 00:38:39.985 align:middle line:84% But this doesn't have any moving parts, I guess. 00:38:39.985 --> 00:38:43.702 align:middle line:84% Does that count as a way to try to make a camera that 00:38:43.702 --> 00:38:45.160 align:middle line:84% perceives depth by having something 00:38:45.160 --> 00:38:47.502 align:middle line:90% that scans physical optics-- 00:38:47.502 --> 00:38:48.460 align:middle line:90% RAMESH RASKAR: Exactly. 00:38:48.460 --> 00:38:48.850 align:middle line:90% AUDIENCE: --around. 00:38:48.850 --> 00:38:50.350 align:middle line:84% RAMESH RASKAR: I think you're asking 00:38:50.350 --> 00:38:51.460 align:middle line:90% a very important question. 00:38:51.460 --> 00:38:52.627 align:middle line:90% Let's see if I have a slide. 00:38:52.627 --> 00:38:56.170 align:middle line:84% So if you think about a different way 00:38:56.170 --> 00:38:57.040 align:middle line:90% of scanning in 3D. 00:38:57.040 --> 00:38:58.390 align:middle line:90% And this is by the way-- 00:38:58.390 --> 00:39:00.460 align:middle line:84% it's one of our visiting students 00:39:00.460 --> 00:39:03.190 align:middle line:90% Doug Lanman and Gabriel Taubin. 00:39:03.190 --> 00:39:05.380 align:middle line:84% They have a beautiful course at SIGGRAPH 00:39:05.380 --> 00:39:09.730 align:middle line:84% on all different ways of doing 3D scanning. 00:39:09.730 --> 00:39:13.330 align:middle line:84% And they're contact based and then non-contact based. 00:39:13.330 --> 00:39:16.390 align:middle line:84% And right now we're talking about mostly active by using 00:39:16.390 --> 00:39:17.600 align:middle line:90% time of flight and so on. 00:39:17.600 --> 00:39:21.460 align:middle line:84% But they're of course passive as well studio and motion 00:39:21.460 --> 00:39:23.700 align:middle line:84% and so on based on focus and defocus. 00:39:23.700 --> 00:39:26.200 align:middle line:84% That may be a good segue way to actually look at this camera 00:39:26.200 --> 00:39:26.930 align:middle line:90% here. 00:39:26.930 --> 00:39:28.690 align:middle line:84% And Ms. Emily, you want to run it? 00:39:28.690 --> 00:39:29.380 align:middle line:90% AUDIENCE: Yeah, [INAUDIBLE] 00:39:29.380 --> 00:39:30.213 align:middle line:90% RAMESH RASKAR: Yeah. 00:39:30.213 --> 00:39:33.190 align:middle line:84% So this camera is completely passive. 00:39:33.190 --> 00:39:36.991 align:middle line:84% It's like a full camera, which has-- 00:39:36.991 --> 00:39:39.010 align:middle line:90% You want to talk about it. 00:39:39.010 --> 00:39:41.390 align:middle line:84% AUDIENCE: Yeah, basically it's 25 separate images, 00:39:41.390 --> 00:39:43.015 align:middle line:84% each one of these is a separate camera. 00:39:43.015 --> 00:39:44.848 align:middle line:84% RAMESH RASKAR: Which is on a single chassis. 00:39:44.848 --> 00:39:47.060 align:middle line:84% AUDIENCE: Yeah, and they're just 25 to 30 minutes. 00:39:47.060 --> 00:39:49.910 align:middle line:90% 00:39:49.910 --> 00:39:53.580 align:middle line:84% But each one's from a slightly different perspective. 00:39:53.580 --> 00:39:55.850 align:middle line:84% So you can do things like add all the images together 00:39:55.850 --> 00:39:58.160 align:middle line:90% but slightly shift them. 00:39:58.160 --> 00:40:03.500 align:middle line:84% So I can focus down at infinity towards the end of the table. 00:40:03.500 --> 00:40:07.250 align:middle line:84% Or I can change the focus by just simply 00:40:07.250 --> 00:40:11.250 align:middle line:84% by shifting or moving the images to focus right there. 00:40:11.250 --> 00:40:13.700 align:middle line:84% RAMESH RASKAR: So remember he's doing all this operation 00:40:13.700 --> 00:40:15.140 align:middle line:90% in software. 00:40:15.140 --> 00:40:16.910 align:middle line:90% You can take those 25 images. 00:40:16.910 --> 00:40:20.070 align:middle line:84% And then in software you can refocus it anywhere you want. 00:40:20.070 --> 00:40:25.640 align:middle line:84% And then, again, based on maximum contrast 00:40:25.640 --> 00:40:27.050 align:middle line:90% or one of those operators. 00:40:27.050 --> 00:40:30.722 align:middle line:84% You could potentially figure out what's in front 00:40:30.722 --> 00:40:31.430 align:middle line:90% or what's behind. 00:40:31.430 --> 00:40:34.830 align:middle line:84% But of course, if you put in some transparent object, 00:40:34.830 --> 00:40:36.830 align:middle line:84% it's going to be quite challenging to figure out 00:40:36.830 --> 00:40:37.820 align:middle line:90% where this is. 00:40:37.820 --> 00:40:41.190 align:middle line:84% If I just put something that's really flat, 00:40:41.190 --> 00:40:43.040 align:middle line:84% then it wouldn't know whether it's in focus 00:40:43.040 --> 00:40:44.900 align:middle line:84% or not in focus in the middle of the paper. 00:40:44.900 --> 00:40:48.890 align:middle line:84% It might be able to do an OK job on the boundaries of the paper. 00:40:48.890 --> 00:40:50.780 align:middle line:84% Or in the middle of the paper it always looks 00:40:50.780 --> 00:40:54.540 align:middle line:90% like it's low frequency. 00:40:54.540 --> 00:40:58.230 align:middle line:84% So these kind of cameras are coming. 00:40:58.230 --> 00:41:03.660 align:middle line:84% You can buy this camera for about $20,000 from U-Plus. 00:41:03.660 --> 00:41:07.720 align:middle line:84% So far we are up to what, 30, up to 30K. 00:41:07.720 --> 00:41:09.870 align:middle line:90% We did 10, 120. 00:41:09.870 --> 00:41:11.190 align:middle line:90% The next one will be-- 00:41:11.190 --> 00:41:14.790 align:middle line:84% we'll get to 60s very quickly, 60K. 00:41:14.790 --> 00:41:20.550 align:middle line:84% And if you, again, think about your cell phone camera, 00:41:20.550 --> 00:41:23.737 align:middle line:84% the current mindset in camera makers 00:41:23.737 --> 00:41:26.070 align:middle line:84% is that they want to shrink it, something that's smaller 00:41:26.070 --> 00:41:28.080 align:middle line:90% and smaller and smaller. 00:41:28.080 --> 00:41:29.970 align:middle line:84% But if you want to create enough baseline, 00:41:29.970 --> 00:41:33.870 align:middle line:84% create some kind of focus based depth extraction, 00:41:33.870 --> 00:41:35.610 align:middle line:84% for example, as we're looking at here, 00:41:35.610 --> 00:41:37.325 align:middle line:84% we must have some baseline between them. 00:41:37.325 --> 00:41:38.700 align:middle line:84% And that's what this camera does. 00:41:38.700 --> 00:41:39.880 align:middle line:90% It's still pretty compact. 00:41:39.880 --> 00:41:40.380 align:middle line:90% I forget. 00:41:40.380 --> 00:41:43.490 align:middle line:84% I think 6 centimeters total, something like that. 00:41:43.490 --> 00:41:44.520 align:middle line:90% I don't know. 00:41:44.520 --> 00:41:46.530 align:middle line:90% It could be on your phone. 00:41:46.530 --> 00:41:48.520 align:middle line:90% It does have that much space. 00:41:48.520 --> 00:41:53.550 align:middle line:84% So if they would just turn it around and make all the pixels 00:41:53.550 --> 00:41:56.670 align:middle line:84% as sensing pixels, you have enough baseline 00:41:56.670 --> 00:41:58.920 align:middle line:84% from the left edge of the camera, 00:41:58.920 --> 00:42:01.410 align:middle line:84% left edge of this cell phone camera 00:42:01.410 --> 00:42:03.870 align:middle line:84% to the right edge of the cell phone camera 00:42:03.870 --> 00:42:05.650 align:middle line:90% to create those effects. 00:42:05.650 --> 00:42:07.350 align:middle line:90% But that mindset has to change. 00:42:07.350 --> 00:42:10.800 align:middle line:84% Right now, as we will see in the class later 00:42:10.800 --> 00:42:17.160 align:middle line:84% on the section on sensors, there is tremendous innovation 00:42:17.160 --> 00:42:19.650 align:middle line:90% in how photons-- 00:42:19.650 --> 00:42:21.780 align:middle line:84% sorry, image sensors are being built 00:42:21.780 --> 00:42:26.340 align:middle line:84% that are wafer level cameras back start elimination, 00:42:26.340 --> 00:42:29.850 align:middle line:90% 3D VLSI, and so on. 00:42:29.850 --> 00:42:32.280 align:middle line:84% I mean, it's all great, and it's going to help us. 00:42:32.280 --> 00:42:36.810 align:middle line:84% But unfortunately all of them have a single tract mind. 00:42:36.810 --> 00:42:38.820 align:middle line:84% They just want to make a higher signal 00:42:38.820 --> 00:42:41.850 align:middle line:84% to noise ratio that must collect as many photons as you can. 00:42:41.850 --> 00:42:45.150 align:middle line:84% And they want to shrink the sensor as small as you can 00:42:45.150 --> 00:42:50.190 align:middle line:84% so that they can sell it for a cheap cost. 00:42:50.190 --> 00:42:52.440 align:middle line:90% But if you think about-- 00:42:52.440 --> 00:42:55.440 align:middle line:84% if somebody had told you 30 years ago 00:42:55.440 --> 00:42:58.720 align:middle line:84% that the TVs are really expensive, 00:42:58.720 --> 00:43:01.230 align:middle line:84% so what we're going to do is start building TVs that 00:43:01.230 --> 00:43:04.950 align:middle line:84% are smaller and smaller because they'll be cheaper and cheaper, 00:43:04.950 --> 00:43:05.910 align:middle line:90% all right-- 00:43:05.910 --> 00:43:07.530 align:middle line:90% but that's not how it works. 00:43:07.530 --> 00:43:10.050 align:middle line:84% People want larger and larger TVs. 00:43:10.050 --> 00:43:14.940 align:middle line:84% And they're willing to pay for 40 inch and 50 inch TVs. 00:43:14.940 --> 00:43:17.370 align:middle line:84% And there will come a time when people say, 00:43:17.370 --> 00:43:20.370 align:middle line:84% it's OK if I have to pay a little bit more, 00:43:20.370 --> 00:43:24.760 align:middle line:84% but make my sensor and make my sensor array larger and larger 00:43:24.760 --> 00:43:29.475 align:middle line:84% so that it's almost the size of the whole device. 00:43:29.475 --> 00:43:33.180 align:middle line:84% Even for a camera like an SLR camera, 00:43:33.180 --> 00:43:35.220 align:middle line:90% not the whole camera is sensor. 00:43:35.220 --> 00:43:37.320 align:middle line:84% Only a small part of it is sensor. 00:43:37.320 --> 00:43:39.240 align:middle line:84% But once we get around this notion 00:43:39.240 --> 00:43:41.190 align:middle line:84% that silicon is really expensive. 00:43:41.190 --> 00:43:44.160 align:middle line:84% And we have to shrink it, and we have to create this wafer. 00:43:44.160 --> 00:43:46.380 align:middle line:84% We can slice and dice it into millions 00:43:46.380 --> 00:43:50.430 align:middle line:90% of tiny sensors for cameras. 00:43:50.430 --> 00:43:52.230 align:middle line:84% Hopefully things will change around 00:43:52.230 --> 00:43:56.100 align:middle line:84% and we won't be watching those tiny TVs as they 00:43:56.100 --> 00:43:57.330 align:middle line:90% have predicted some time ago. 00:43:57.330 --> 00:44:00.220 align:middle line:90% 00:44:00.220 --> 00:44:03.790 align:middle line:84% So that's kind of where some of these things are. 00:44:03.790 --> 00:44:07.010 align:middle line:90% So let me go back-- 00:44:07.010 --> 00:44:11.050 align:middle line:84% so getting 3D very challenging, and that I would say 00:44:11.050 --> 00:44:14.980 align:middle line:84% is kind of version 0.1 of a computation camera. 00:44:14.980 --> 00:44:17.725 align:middle line:84% I mean, it has to sense the three-dimensional world 00:44:17.725 --> 00:44:20.830 align:middle line:90% to do anything interesting. 00:44:20.830 --> 00:44:23.020 align:middle line:84% And it is being used in other scenarios, 00:44:23.020 --> 00:44:25.790 align:middle line:90% I mean, [INAUDIBLE] challenge. 00:44:25.790 --> 00:44:28.730 align:middle line:84% This is an animation for movies.-- 00:44:28.730 --> 00:44:33.190 align:middle line:84% where you can build vehicles that 00:44:33.190 --> 00:44:38.920 align:middle line:84% can navigate through any kind of terrain, 00:44:38.920 --> 00:44:42.250 align:middle line:90% including urban neighborhoods. 00:44:42.250 --> 00:44:44.560 align:middle line:84% So the first version of the [INAUDIBLE] challenge 00:44:44.560 --> 00:44:47.260 align:middle line:90% actually had a lot of cameras. 00:44:47.260 --> 00:44:50.590 align:middle line:84% But the sad aspect from a camera point of view 00:44:50.590 --> 00:44:55.120 align:middle line:84% is that the second version that was-- 00:44:55.120 --> 00:44:58.810 align:middle line:84% Stanford was the winner in that one. 00:44:58.810 --> 00:45:00.850 align:middle line:84% In the traditional definition of a camera, 00:45:00.850 --> 00:45:04.690 align:middle line:84% there were no cameras on the whole vehicle. 00:45:04.690 --> 00:45:08.230 align:middle line:84% There were zero cameras on a car that's 00:45:08.230 --> 00:45:11.883 align:middle line:90% built to navigate in a city. 00:45:11.883 --> 00:45:12.550 align:middle line:90% It's pretty sad. 00:45:12.550 --> 00:45:17.050 align:middle line:84% I mean, if you think about it, a human driver driving 00:45:17.050 --> 00:45:18.610 align:middle line:90% through a city-- 00:45:18.610 --> 00:45:21.850 align:middle line:84% I would say almost all our input, 00:45:21.850 --> 00:45:25.570 align:middle line:84% all our actions are based on visual information. 00:45:25.570 --> 00:45:31.270 align:middle line:84% But cameras are so primitive to build this self navigating 00:45:31.270 --> 00:45:33.730 align:middle line:84% vehicle that no traditional cameras are used. 00:45:33.730 --> 00:45:35.650 align:middle line:84% Now, of course, I'm taking it to the extreme 00:45:35.650 --> 00:45:37.150 align:middle line:84% because the kind of devices they did 00:45:37.150 --> 00:45:40.900 align:middle line:84% use were similar to range scanning devices. 00:45:40.900 --> 00:45:44.410 align:middle line:84% They had laser scanners LIDAR, reducing time of flight 00:45:44.410 --> 00:45:45.200 align:middle line:90% and so on. 00:45:45.200 --> 00:45:48.340 align:middle line:84% And in a way they're capturing information 00:45:48.340 --> 00:45:50.750 align:middle line:84% that's sensitive to different directions. 00:45:50.750 --> 00:45:53.350 align:middle line:84% So in a way, it's a camera, but it's 00:45:53.350 --> 00:45:57.370 align:middle line:84% not a traditional visible range camera that they're using. 00:45:57.370 --> 00:45:59.740 align:middle line:84% It's a detector, a single pixel detector 00:45:59.740 --> 00:46:04.687 align:middle line:84% that's measuring light coming from different directions. 00:46:04.687 --> 00:46:06.520 align:middle line:84% And on the other extreme, that's one extreme 00:46:06.520 --> 00:46:09.220 align:middle line:84% where cameras are not good enough to self navigate. 00:46:09.220 --> 00:46:13.630 align:middle line:84% On the other extreme, you have lots of people online-- 00:46:13.630 --> 00:46:19.340 align:middle line:84% again, this is a slide from Doug Lanman building their own 3D 00:46:19.340 --> 00:46:19.840 align:middle line:90% scanners. 00:46:19.840 --> 00:46:22.600 align:middle line:84% So of course, you can take a Logitech webcam 00:46:22.600 --> 00:46:23.890 align:middle line:90% and put some [INAUDIBLE]. 00:46:23.890 --> 00:46:26.590 align:middle line:84% You can calibrate that and move it around 00:46:26.590 --> 00:46:31.630 align:middle line:84% to create 3D models and a whole bunch of-- 00:46:31.630 --> 00:46:33.820 align:middle line:90% We'll look at this during the-- 00:46:33.820 --> 00:46:37.360 align:middle line:84% people are using just a wineglass 00:46:37.360 --> 00:46:38.860 align:middle line:90% to create laser stripes. 00:46:38.860 --> 00:46:42.940 align:middle line:84% So I can take a laser pointer and shine it at a wineglass 00:46:42.940 --> 00:46:44.290 align:middle line:90% to create a laser stripe. 00:46:44.290 --> 00:46:46.300 align:middle line:84% I don't even have to buy expensive optics. 00:46:46.300 --> 00:46:49.390 align:middle line:84% And they just scan it and develop a cheap DV 00:46:49.390 --> 00:46:51.550 align:middle line:90% Cam to create 3D scanners. 00:46:51.550 --> 00:46:54.010 align:middle line:84% This one is probably most interesting. 00:46:54.010 --> 00:46:56.710 align:middle line:84% They want to create, again, a very accurate 00:46:56.710 --> 00:46:59.650 align:middle line:90% 3D model from a LEGO rig. 00:46:59.650 --> 00:47:02.028 align:middle line:90% And this is how it works. 00:47:02.028 --> 00:47:03.820 align:middle line:84% You can probably guess it from the picture. 00:47:03.820 --> 00:47:05.500 align:middle line:90% There is a milk pot. 00:47:05.500 --> 00:47:07.480 align:middle line:84% They want to take a character and scan it. 00:47:07.480 --> 00:47:10.120 align:middle line:84% They'll put this character in a milk pot. 00:47:10.120 --> 00:47:11.830 align:middle line:84% And take a photo from above doing 00:47:11.830 --> 00:47:13.090 align:middle line:90% a very simple segmentation. 00:47:13.090 --> 00:47:14.620 align:middle line:84% And then over time, they're going 00:47:14.620 --> 00:47:21.100 align:middle line:84% to put a little bit more milk so that the level of the fluid 00:47:21.100 --> 00:47:25.540 align:middle line:84% will rise in this square bucket and this rectangular bucket. 00:47:25.540 --> 00:47:27.100 align:middle line:84% And they'll continue to take pictures 00:47:27.100 --> 00:47:31.840 align:middle line:84% and they'll create this 3D model section by section. 00:47:31.840 --> 00:47:32.618 align:middle line:90% Pretty amazing. 00:47:32.618 --> 00:47:34.535 align:middle line:84% AUDIENCE: They did it in New York with humans. 00:47:34.535 --> 00:47:35.410 align:middle line:90% RAMESH RASKAR: Sorry? 00:47:35.410 --> 00:47:37.720 align:middle line:84% AUDIENCE: They did it in New York with humans. 00:47:37.720 --> 00:47:40.380 align:middle line:84% They go in the milk and scan you. 00:47:40.380 --> 00:47:42.372 align:middle line:90% [INAUDIBLE] 00:47:42.372 --> 00:47:44.080 align:middle line:84% RAMESH RASKAR: Excellent, I would imagine 00:47:44.080 --> 00:47:45.330 align:middle line:90% going the other way is easier. 00:47:45.330 --> 00:47:47.620 align:middle line:90% Just drink. 00:47:47.620 --> 00:47:49.650 align:middle line:84% Just straight up [INAUDIBLE] and drain it. 00:47:49.650 --> 00:47:51.490 align:middle line:84% It will go out almost concentrated. 00:47:51.490 --> 00:47:53.710 align:middle line:90% AUDIENCE: True. 00:47:53.710 --> 00:47:56.560 align:middle line:84% RAMESH RASKAR: But, yeah, this could be a great class project. 00:47:56.560 --> 00:48:02.470 align:middle line:90% 00:48:02.470 --> 00:48:05.890 align:middle line:84% So here's a slightly different direction 00:48:05.890 --> 00:48:07.300 align:middle line:90% that some people are taking. 00:48:07.300 --> 00:48:11.180 align:middle line:84% And I really like this work from [INAUDIBLE] Washington. 00:48:11.180 --> 00:48:12.290 align:middle line:90% Some of you are-- 00:48:12.290 --> 00:48:15.850 align:middle line:84% OK, let me state the question first and see if anyone of you 00:48:15.850 --> 00:48:17.020 align:middle line:90% has an answer to that. 00:48:17.020 --> 00:48:18.970 align:middle line:84% Let's say you go to Rome, and you are 00:48:18.970 --> 00:48:21.880 align:middle line:90% in front of the Trevi Fountain. 00:48:21.880 --> 00:48:24.820 align:middle line:84% You take a photo, and you don't know anything 00:48:24.820 --> 00:48:27.970 align:middle line:84% about Trevi Fountain and all the hype about it. 00:48:27.970 --> 00:48:30.010 align:middle line:84% You want to take a photo and figure out 00:48:30.010 --> 00:48:33.220 align:middle line:84% which part of this photo is interesting 00:48:33.220 --> 00:48:35.630 align:middle line:90% according to everybody else. 00:48:35.630 --> 00:48:37.360 align:middle line:90% So why am I here? 00:48:37.360 --> 00:48:39.100 align:middle line:90% Should I be looking at-- 00:48:39.100 --> 00:48:42.490 align:middle line:84% if I go to the Old Town Square in Prague, 00:48:42.490 --> 00:48:44.350 align:middle line:84% I'm not going to look at a fountain here. 00:48:44.350 --> 00:48:47.950 align:middle line:84% I'm going to look at another part of the castle. 00:48:47.950 --> 00:48:49.990 align:middle line:84% But somehow when I'm at Trevi Fountain, 00:48:49.990 --> 00:48:51.490 align:middle line:84% I'm not looking at the buildings. 00:48:51.490 --> 00:48:52.840 align:middle line:90% I'm looking at the Fountain. 00:48:52.840 --> 00:48:58.150 align:middle line:84% So how would I determine without looking in my guidebook which 00:48:58.150 --> 00:48:59.956 align:middle line:90% part is most interesting? 00:48:59.956 --> 00:49:03.150 align:middle line:90% 00:49:03.150 --> 00:49:04.140 align:middle line:90% Any clues, any answers? 00:49:04.140 --> 00:49:13.310 align:middle line:90% 00:49:13.310 --> 00:49:15.060 align:middle line:84% AUDIENCE: I don't know if I should answer. 00:49:15.060 --> 00:49:16.060 align:middle line:90% RAMESH RASKAR: Go ahead. 00:49:16.060 --> 00:49:17.360 align:middle line:90% You're [INAUDIBLE] yeah. 00:49:17.360 --> 00:49:19.550 align:middle line:84% AUDIENCE: One thing you could perhaps do 00:49:19.550 --> 00:49:21.800 align:middle line:84% is take a consensus for all the photos 00:49:21.800 --> 00:49:24.040 align:middle line:84% that have been taken of this object 00:49:24.040 --> 00:49:27.745 align:middle line:84% and basically see where the feature tracks overlap. 00:49:27.745 --> 00:49:31.080 align:middle line:90% 00:49:31.080 --> 00:49:33.788 align:middle line:90% An alert [INAUDIBLE] sort of. 00:49:33.788 --> 00:49:34.830 align:middle line:90% RAMESH RASKAR: Excellent. 00:49:34.830 --> 00:49:39.960 align:middle line:84% So this is actually an offshoot of the photo tourism Microsoft 00:49:39.960 --> 00:49:43.020 align:middle line:84% Photosynth project where you have millions of photos 00:49:43.020 --> 00:49:45.760 align:middle line:90% of the same tourist location. 00:49:45.760 --> 00:49:49.380 align:middle line:84% And once you have registered them in 3D, 00:49:49.380 --> 00:49:51.240 align:middle line:84% then you can just shoot the rays back 00:49:51.240 --> 00:49:54.390 align:middle line:84% to see which rays will intersect the pixels. 00:49:54.390 --> 00:49:59.760 align:middle line:84% If most of the rays seem to be shooting you just kind of just 00:49:59.760 --> 00:50:03.120 align:middle line:84% a histogram, you'll realize that most of the photos 00:50:03.120 --> 00:50:04.420 align:middle line:90% are looking at the fountain. 00:50:04.420 --> 00:50:05.795 align:middle line:84% And in this case, what the photos 00:50:05.795 --> 00:50:08.010 align:middle line:84% are looking up at the top part of the [INAUDIBLE].. 00:50:08.010 --> 00:50:09.960 align:middle line:84% AUDIENCE: So maybe it's what is popular here. 00:50:09.960 --> 00:50:11.640 align:middle line:84% Everything might be a different [INAUDIBLE].. 00:50:11.640 --> 00:50:13.390 align:middle line:84% RAMESH RASKAR: You're right, you're right. 00:50:13.390 --> 00:50:14.940 align:middle line:84% So the next question is, how can we-- 00:50:14.940 --> 00:50:17.500 align:middle line:84% it's like popular versus interesting on Flickr. 00:50:17.500 --> 00:50:20.550 align:middle line:90% AUDIENCE: Yes, that's true. 00:50:20.550 --> 00:50:23.550 align:middle line:84% RAMESH RASKAR: So here's our question, how can we ask-- 00:50:23.550 --> 00:50:25.135 align:middle line:90% how can we answer this question? 00:50:25.135 --> 00:50:26.760 align:middle line:84% AUDIENCE: There may be certain features 00:50:26.760 --> 00:50:30.240 align:middle line:84% that the human visual system finds particularly interesting, 00:50:30.240 --> 00:50:32.730 align:middle line:84% things with high levels of non-repetitive detail 00:50:32.730 --> 00:50:36.028 align:middle line:90% or certain shapes or ratios. 00:50:36.028 --> 00:50:37.320 align:middle line:90% RAMESH RASKAR: Yeah, certainly. 00:50:37.320 --> 00:50:39.670 align:middle line:84% So I think there's a lot to be done. 00:50:39.670 --> 00:50:43.320 align:middle line:84% So maybe if we want to build a camera that directly detects 00:50:43.320 --> 00:50:45.720 align:middle line:84% the building pattern, it doesn't care about finding it, 00:50:45.720 --> 00:50:47.160 align:middle line:90% capturing a real photo. 00:50:47.160 --> 00:50:49.860 align:middle line:84% But it does a very good job of finding out 00:50:49.860 --> 00:50:52.440 align:middle line:84% if there are patterns we find interesting whether it's 00:50:52.440 --> 00:50:58.230 align:middle line:84% symmetry or repetition or the right scales the right aspect 00:50:58.230 --> 00:50:59.520 align:middle line:90% ratio and so on. 00:50:59.520 --> 00:51:04.700 align:middle line:90% 00:51:04.700 --> 00:51:07.550 align:middle line:90% And the same project. 00:51:07.550 --> 00:51:10.350 align:middle line:84% So this is the Pantheon again in Rome. 00:51:10.350 --> 00:51:13.370 align:middle line:84% These are all the places from which people take photos. 00:51:13.370 --> 00:51:17.870 align:middle line:84% And if I just tell you this is the view of the Pantheon, 00:51:17.870 --> 00:51:21.320 align:middle line:84% and you usually are-- up here is a fountain. 00:51:21.320 --> 00:51:24.800 align:middle line:84% People start from here and then go in and roam around 00:51:24.800 --> 00:51:28.940 align:middle line:84% and take a picture of this big hole in the Pantheon. 00:51:28.940 --> 00:51:34.010 align:middle line:84% So the question is if I take a picture here at the entrance 00:51:34.010 --> 00:51:37.100 align:middle line:84% and from inside the pantheon looking out 00:51:37.100 --> 00:51:39.860 align:middle line:84% at the door and a picture from outside, 00:51:39.860 --> 00:51:44.330 align:middle line:84% what's the part that would connect these two photos? 00:51:44.330 --> 00:51:45.740 align:middle line:90% It's not a straight line path. 00:51:45.740 --> 00:51:49.023 align:middle line:90% 00:51:49.023 --> 00:51:49.940 align:middle line:90% I don't have an image. 00:51:49.940 --> 00:51:52.770 align:middle line:90% 00:51:52.770 --> 00:51:55.600 align:middle line:90% I don't have it. 00:51:55.600 --> 00:51:59.038 align:middle line:84% But if you actually have this, again, voting scheme, 00:51:59.038 --> 00:52:01.080 align:middle line:84% you've realized the best way to go from this view 00:52:01.080 --> 00:52:04.810 align:middle line:84% to this view is follow this particular path. 00:52:04.810 --> 00:52:10.200 align:middle line:84% So this is data that's been captured from visual media, 00:52:10.200 --> 00:52:13.380 align:middle line:84% like photographs, but inherently it's more about geometry. 00:52:13.380 --> 00:52:14.280 align:middle line:90% And I would call it-- 00:52:14.280 --> 00:52:16.890 align:middle line:90% it's almost non-visual. 00:52:16.890 --> 00:52:18.750 align:middle line:84% So that was computational camera, 00:52:18.750 --> 00:52:22.680 align:middle line:84% and let's think about computational photography. 00:52:22.680 --> 00:52:29.190 align:middle line:84% And computational photography and my collaborator Jack 00:52:29.190 --> 00:52:31.590 align:middle line:90% Tumblin at Northwestern. 00:52:31.590 --> 00:52:33.780 align:middle line:84% This is how we like to define the two parts. 00:52:33.780 --> 00:52:36.100 align:middle line:84% We want to capture the scene, and we 00:52:36.100 --> 00:52:37.350 align:middle line:90% want to synthesize this thing. 00:52:37.350 --> 00:52:40.080 align:middle line:84% And when we capture, we want to capture it 00:52:40.080 --> 00:52:42.648 align:middle line:84% in an extremely rich fashion so that it's machine readable 00:52:42.648 --> 00:52:44.190 align:middle line:84% and the machine can understand what's 00:52:44.190 --> 00:52:46.560 align:middle line:90% out there and re-synthesize. 00:52:46.560 --> 00:52:49.080 align:middle line:84% We want to synthesize in a hyper realistic manner 00:52:49.080 --> 00:52:52.770 align:middle line:84% so that it represents the essence 00:52:52.770 --> 00:52:55.930 align:middle line:90% of our visual experience. 00:52:55.930 --> 00:52:59.380 align:middle line:84% And within that there are three major teams. 00:52:59.380 --> 00:53:03.880 align:middle line:84% One is epsilon photography, which is basically generalized 00:53:03.880 --> 00:53:04.380 align:middle line:90% bracketing. 00:53:04.380 --> 00:53:08.610 align:middle line:84% So whether I want HDR or panorama, 00:53:08.610 --> 00:53:11.760 align:middle line:84% I'm going to take my camera, Google's exposure bracketing 00:53:11.760 --> 00:53:15.000 align:middle line:84% or focus bracketing or view bracketing 00:53:15.000 --> 00:53:18.240 align:middle line:84% and so on and just create a very nice picture 00:53:18.240 --> 00:53:21.510 align:middle line:84% and mostly to overcome the limitations of a camera. 00:53:21.510 --> 00:53:22.920 align:middle line:90% That's just epsilon photography. 00:53:22.920 --> 00:53:25.380 align:middle line:84% I'm going to change the parameters of the camera 00:53:25.380 --> 00:53:29.340 align:middle line:90% within an epsilon neighborhood. 00:53:29.340 --> 00:53:31.980 align:middle line:84% And then people think of that as the ultimate camera. 00:53:31.980 --> 00:53:34.860 align:middle line:84% But again we're not going to focus much on this part 00:53:34.860 --> 00:53:36.510 align:middle line:90% in this course. 00:53:36.510 --> 00:53:38.580 align:middle line:84% The next part-- that's quite interesting-- 00:53:38.580 --> 00:53:40.960 align:middle line:90% is so-called coated photography. 00:53:40.960 --> 00:53:44.250 align:middle line:84% So the comment we had earlier I really care 00:53:44.250 --> 00:53:47.100 align:middle line:90% about some mid-level feature. 00:53:47.100 --> 00:53:48.690 align:middle line:84% I want to know if they're symmetric. 00:53:48.690 --> 00:53:50.610 align:middle line:84% I want to if there's a repeating pattern. 00:53:50.610 --> 00:53:53.100 align:middle line:84% I want to know where the edges are. 00:53:53.100 --> 00:53:56.460 align:middle line:84% I want to know how the regions can be segmented and so on. 00:53:56.460 --> 00:53:59.310 align:middle line:84% And it's worth taking multiple photos such as in bracketing. 00:53:59.310 --> 00:54:03.120 align:middle line:84% I just want to take one or maybe two photos which are reversible 00:54:03.120 --> 00:54:07.740 align:middle line:84% and of course the information of what the world in my image. 00:54:07.740 --> 00:54:10.390 align:middle line:84% And the image may not be a single 2D image. 00:54:10.390 --> 00:54:12.110 align:middle line:84% It could be a light field camera where 00:54:12.110 --> 00:54:15.090 align:middle line:84% you have 25 images or a single snapshot 00:54:15.090 --> 00:54:16.890 align:middle line:84% or you could have a time of flight camera 00:54:16.890 --> 00:54:19.020 align:middle line:84% where you really can't call it a photo 00:54:19.020 --> 00:54:23.310 align:middle line:84% because it's actually measuring the amount of time 00:54:23.310 --> 00:54:26.130 align:middle line:84% it takes for light to travel back and forth in a given 00:54:26.130 --> 00:54:29.280 align:middle line:90% direction and so on. 00:54:29.280 --> 00:54:31.960 align:middle line:84% And it would be very useful for scene analysis. 00:54:31.960 --> 00:54:37.740 align:middle line:84% So a lot of our course would be actually discussing this part. 00:54:37.740 --> 00:54:39.780 align:middle line:84% And the last part is [? instance ?] photography 00:54:39.780 --> 00:54:42.990 align:middle line:84% which is really to see if we can go 00:54:42.990 --> 00:54:47.520 align:middle line:84% beyond this low level of features, pixels and mid-level 00:54:47.520 --> 00:54:51.150 align:middle line:84% features such as motion and foreground and background 00:54:51.150 --> 00:54:55.500 align:middle line:84% and symmetry and so on to a higher level understanding. 00:54:55.500 --> 00:54:57.900 align:middle line:84% So that it's not just mimicking human eye, 00:54:57.900 --> 00:55:01.830 align:middle line:84% but it's doing possibly things like this, 00:55:01.830 --> 00:55:05.070 align:middle line:84% telling me what's popular for example. 00:55:05.070 --> 00:55:08.260 align:middle line:90% 00:55:08.260 --> 00:55:13.940 align:middle line:84% And I claim that only when we have computational cameras 00:55:13.940 --> 00:55:16.710 align:middle line:84% and computational photography supporting this, 00:55:16.710 --> 00:55:19.535 align:middle line:84% we'll be able to create new visual art forms. 00:55:19.535 --> 00:55:22.800 align:middle line:90% 00:55:22.800 --> 00:55:27.720 align:middle line:84% So within that, this is a chart that we'll 00:55:27.720 --> 00:55:29.520 align:middle line:84% be talking about throughout this class. 00:55:29.520 --> 00:55:31.380 align:middle line:84% I won't go into too much detail right now. 00:55:31.380 --> 00:55:34.080 align:middle line:84% But we have certain goals, the mid-level features, 00:55:34.080 --> 00:55:36.820 align:middle line:84% low level features, high level features, and so on. 00:55:36.820 --> 00:55:40.800 align:middle line:84% And we have certain tools that can capture the raw image. 00:55:40.800 --> 00:55:44.280 align:middle line:84% We can capture the incident angle and spectrum 00:55:44.280 --> 00:55:47.190 align:middle line:90% such as UV and thermal IR. 00:55:47.190 --> 00:55:49.650 align:middle line:84% We can capture high dimensional reflectance field. 00:55:49.650 --> 00:55:54.660 align:middle line:84% We have non-visual data such as GPS and identity, metadata, 00:55:54.660 --> 00:55:57.190 align:middle line:90% image priors and so on. 00:55:57.190 --> 00:56:00.120 align:middle line:84% And with that we're going to explore this whole space 00:56:00.120 --> 00:56:02.790 align:middle line:90% of camera and photography. 00:56:02.790 --> 00:56:06.120 align:middle line:84% Of course high dynamic range and so on is right here. 00:56:06.120 --> 00:56:08.110 align:middle line:84% We're not going to spend too much time on that. 00:56:08.110 --> 00:56:09.818 align:middle line:84% But we're going to think about how can we 00:56:09.818 --> 00:56:12.420 align:middle line:84% insert a virtual object, how can you take a photo 00:56:12.420 --> 00:56:18.210 align:middle line:84% and relight, change the lighting in the scene and so on. 00:56:18.210 --> 00:56:22.440 align:middle line:84% Material editing for example from a single photon, Ted Olson 00:56:22.440 --> 00:56:27.400 align:middle line:84% is here, and he's done a lot of work in this space. 00:56:27.400 --> 00:56:31.560 align:middle line:84% And if you look at this chart, you 00:56:31.560 --> 00:56:36.540 align:middle line:84% realize that even the human vision is not 00:56:36.540 --> 00:56:39.900 align:middle line:84% at the top right of this diagram. 00:56:39.900 --> 00:56:43.290 align:middle line:84% With human vision I cannot look around the corner. 00:56:43.290 --> 00:56:45.040 align:middle line:84% I cannot see what's inside the body. 00:56:45.040 --> 00:56:47.820 align:middle line:84% I cannot tell you what's behind the curtain. 00:56:47.820 --> 00:56:52.530 align:middle line:84% I cannot tell you when I'm in Rome what's interesting 00:56:52.530 --> 00:56:53.470 align:middle line:90% and so on. 00:56:53.470 --> 00:56:56.790 align:middle line:84% So you really want to create this augmented human experience 00:56:56.790 --> 00:57:01.830 align:middle line:84% by using these tools and using the mechanisms that we 00:57:01.830 --> 00:57:02.650 align:middle line:90% have available. 00:57:02.650 --> 00:57:04.860 align:middle line:90% So we'll be coming back to this. 00:57:04.860 --> 00:57:05.910 align:middle line:90% Just keep this in mind. 00:57:05.910 --> 00:57:08.670 align:middle line:90% 00:57:08.670 --> 00:57:11.790 align:middle line:84% And a lot of this is actually available in the book 00:57:11.790 --> 00:57:15.390 align:middle line:84% that I and Jack Tumblin have published. 00:57:15.390 --> 00:57:17.430 align:middle line:84% The PDF of this book will be available 00:57:17.430 --> 00:57:18.420 align:middle line:90% throughout the course. 00:57:18.420 --> 00:57:20.520 align:middle line:84% And the real book should be out any time. 00:57:20.520 --> 00:57:24.440 align:middle line:90% 00:57:24.440 --> 00:57:25.230 align:middle line:90% All right. 00:57:25.230 --> 00:57:29.603 align:middle line:84% So just a couple of more slides, and we'll take a break. 00:57:29.603 --> 00:57:31.020 align:middle line:84% Let me skip over this one for now, 00:57:31.020 --> 00:57:34.580 align:middle line:90% some of my favorite examples. 00:57:34.580 --> 00:57:35.240 align:middle line:90% All right. 00:57:35.240 --> 00:57:38.540 align:middle line:84% So we also got to spend a lot of time thinking 00:57:38.540 --> 00:57:44.750 align:middle line:84% about cameras that, again, show what's not seen. 00:57:44.750 --> 00:57:47.540 align:middle line:84% And I realized that Mathias is here. 00:57:47.540 --> 00:57:51.650 align:middle line:84% His company is the founder of Redshift, a thermal imaging 00:57:51.650 --> 00:57:52.820 align:middle line:90% company. 00:57:52.820 --> 00:57:56.660 align:middle line:84% And you want to say a couple of words? 00:57:56.660 --> 00:57:58.040 align:middle line:90% MATHIAS OMOTOLA: Sure. 00:57:58.040 --> 00:58:01.160 align:middle line:84% So this was my second startup, and the idea 00:58:01.160 --> 00:58:04.700 align:middle line:84% was to build an ultra low cost thermal imaging camera 00:58:04.700 --> 00:58:08.450 align:middle line:84% and do it using a standard cell phone 00:58:08.450 --> 00:58:10.820 align:middle line:84% camera as, actually, the image sensor 00:58:10.820 --> 00:58:17.360 align:middle line:84% and using a little thermally tunable film as a translator 00:58:17.360 --> 00:58:19.582 align:middle line:90% between infrared and visible. 00:58:19.582 --> 00:58:20.540 align:middle line:90% RAMESH RASKAR: Exactly. 00:58:20.540 --> 00:58:22.700 align:middle line:84% And the goal is to dramatically bring down 00:58:22.700 --> 00:58:26.570 align:middle line:90% the cost of a thermal camera. 00:58:26.570 --> 00:58:30.071 align:middle line:90% So let's bring in our next toy. 00:58:30.071 --> 00:58:31.520 align:middle line:84% Will you need some time to set up? 00:58:31.520 --> 00:58:33.020 align:middle line:90% GUEST SPEAKER: Oh, yes I need-- 00:58:33.020 --> 00:58:33.840 align:middle line:90% RAMESH RASKAR: OK. 00:58:33.840 --> 00:58:34.580 align:middle line:90% Yeah. 00:58:34.580 --> 00:58:38.210 align:middle line:84% I we'll set up this other time, which will very quickly take us 00:58:38.210 --> 00:58:41.390 align:middle line:90% towards the 60K. 00:58:41.390 --> 00:58:43.840 align:middle line:84% And you can do some amazing things. 00:58:43.840 --> 00:58:47.660 align:middle line:84% So this is not from Mathias's company. 00:58:47.660 --> 00:58:50.930 align:middle line:90% Is that a maker or just a-- 00:58:50.930 --> 00:58:54.320 align:middle line:84% they actually make cameras or they're just integrators? 00:58:54.320 --> 00:58:57.020 align:middle line:84% And they can use it in sports analysis? 00:58:57.020 --> 00:59:00.890 align:middle line:84% This is in the game of cricket where you want to figure out 00:59:00.890 --> 00:59:04.190 align:middle line:84% if the ball was hitting the pad or the pad 00:59:04.190 --> 00:59:07.130 align:middle line:84% and it's a very difficult call for the referee 00:59:07.130 --> 00:59:11.750 align:middle line:84% but if they have a thermal imaging camera, then 00:59:11.750 --> 00:59:14.510 align:middle line:84% the ball here if it hits the bag, 00:59:14.510 --> 00:59:16.612 align:middle line:84% it will leave a hot spot on the bat. 00:59:16.612 --> 00:59:19.070 align:middle line:84% But if it hits the bat, it will leave a hotspot on the pad. 00:59:19.070 --> 00:59:20.900 align:middle line:84% So just by looking at an image a couple 00:59:20.900 --> 00:59:24.350 align:middle line:84% of seconds after the ball has left, the player, 00:59:24.350 --> 00:59:28.170 align:middle line:84% you can figure out if the ball hit the-- 00:59:28.170 --> 00:59:31.910 align:middle line:90% the ball hit the bat or the bat. 00:59:31.910 --> 00:59:33.080 align:middle line:90% And this is major. 00:59:33.080 --> 00:59:35.870 align:middle line:84% People get really angry and all the sports analysts 00:59:35.870 --> 00:59:44.420 align:middle line:84% will be writing or the referee next day in the press. 00:59:44.420 --> 00:59:47.090 align:middle line:90% This really saves their life. 00:59:47.090 --> 00:59:49.910 align:middle line:84% But the goal for Mathias and other companies 00:59:49.910 --> 00:59:53.720 align:middle line:84% is to make it available possibly at the same cost 00:59:53.720 --> 00:59:55.400 align:middle line:90% as your traditional camera. 00:59:55.400 --> 00:59:58.130 align:middle line:84% I mean, if you think about a cost of a digital SLR, 00:59:58.130 --> 01:00:00.080 align:middle line:90% it was $30,000 dollars. 01:00:00.080 --> 01:00:03.110 align:middle line:90% And now it's worth about $500. 01:00:03.110 --> 01:00:04.160 align:middle line:90% It's just silicon. 01:00:04.160 --> 01:00:05.570 align:middle line:90% It's not that expensive. 01:00:05.570 --> 01:00:10.460 align:middle line:84% If Canon or Nokia decides to have 01:00:10.460 --> 01:00:12.680 align:middle line:84% a thermal camera in every cell phone, 01:00:12.680 --> 01:00:16.790 align:middle line:84% they will be available for very low cost, very extreme low 01:00:16.790 --> 01:00:17.378 align:middle line:90% cost. 01:00:17.378 --> 01:00:18.170 align:middle line:90% What is your guess? 01:00:18.170 --> 01:00:21.080 align:middle line:84% What would be the cost in five years? 01:00:21.080 --> 01:00:23.350 align:middle line:90% AUDIENCE: Five years? 01:00:23.350 --> 01:00:24.890 align:middle line:90% It's still $500. 01:00:24.890 --> 01:00:25.740 align:middle line:90% RAMESH RASKAR: $500. 01:00:25.740 --> 01:00:27.230 align:middle line:90% That's not bad, right? 01:00:27.230 --> 01:00:29.690 align:middle line:84% And as you'll see later in the class 01:00:29.690 --> 01:00:31.560 align:middle line:90% and you'll see in this demo-- 01:00:31.560 --> 01:00:33.060 align:middle line:84% we can do this in the break as well. 01:00:33.060 --> 01:00:35.930 align:middle line:90% So take your time. 01:00:35.930 --> 01:00:37.940 align:middle line:90% It can do some amazing things. 01:00:37.940 --> 01:00:38.600 align:middle line:90% All right. 01:00:38.600 --> 01:00:43.280 align:middle line:84% So that was computational camera and computational photography. 01:00:43.280 --> 01:00:46.130 align:middle line:84% And in this class we're going to look at both aspects. 01:00:46.130 --> 01:00:48.680 align:middle line:84% This is how we're going to do it. 01:00:48.680 --> 01:00:52.430 align:middle line:84% We have two numbers 131 for undergraduate and 531 01:00:52.430 --> 01:00:54.080 align:middle line:90% for graduate. 01:00:54.080 --> 01:00:57.080 align:middle line:84% The main difference is we have four assignments 01:00:57.080 --> 01:00:59.630 align:middle line:84% for the graduate version and three assignments 01:00:59.630 --> 01:01:01.670 align:middle line:90% for the undergraduate version. 01:01:01.670 --> 01:01:04.220 align:middle line:84% And we have a midterm exam where there 01:01:04.220 --> 01:01:07.700 align:middle line:84% will be fewer questions for the undergraduate version other 01:01:07.700 --> 01:01:11.670 align:middle line:84% than that, the rest of the course is very similar. 01:01:11.670 --> 01:01:15.620 align:middle line:84% So in the assignments we're playing with hands 01:01:15.620 --> 01:01:20.280 align:middle line:84% on with optics and illumination and sensors and other elements. 01:01:20.280 --> 01:01:23.450 align:middle line:84% And we'll have all these toys available for you. 01:01:23.450 --> 01:01:28.520 align:middle line:84% We'll have projectors, different light sources, lasers, 01:01:28.520 --> 01:01:32.000 align:middle line:84% and things like that to play with. 01:01:32.000 --> 01:01:35.150 align:middle line:84% I will also have a best project award. 01:01:35.150 --> 01:01:38.060 align:middle line:84% At the end last year there was exactly 01:01:38.060 --> 01:01:41.330 align:middle line:84% one undergraduate student, and he 01:01:41.330 --> 01:01:44.990 align:middle line:84% won the best project award, pretty impressive, 01:01:44.990 --> 01:01:46.470 align:middle line:90% pretty impressive. 01:01:46.470 --> 01:01:48.560 align:middle line:84% He did an amazing job because last year we did not 01:01:48.560 --> 01:01:50.480 align:middle line:90% have an undergraduate version. 01:01:50.480 --> 01:01:54.440 align:middle line:90% It was only a graduate version. 01:01:54.440 --> 01:01:59.610 align:middle line:84% Midterm exam will be early November. 01:01:59.610 --> 01:02:03.170 align:middle line:84% A big component of the class and your work 01:02:03.170 --> 01:02:06.950 align:middle line:84% would be a final project, which should be novel as well as cool 01:02:06.950 --> 01:02:08.960 align:middle line:90% unlike a lot of other fields. 01:02:08.960 --> 01:02:10.940 align:middle line:84% It is possible to come up with an idea 01:02:10.940 --> 01:02:14.640 align:middle line:84% that nobody else has even thought about in this field. 01:02:14.640 --> 01:02:17.570 align:middle line:84% So you could come up with ideas that are not just incremental 01:02:17.570 --> 01:02:19.340 align:middle line:90% but that could be game changing. 01:02:19.340 --> 01:02:25.700 align:middle line:84% And I'll tell you later that three projects, 01:02:25.700 --> 01:02:30.800 align:middle line:84% final projects from last year led to SIGGRAPH or ICCP papers. 01:02:30.800 --> 01:02:34.970 align:middle line:84% And two of the projects are becoming multi hundred thousand 01:02:34.970 --> 01:02:37.470 align:middle line:90% dollar projects. 01:02:37.470 --> 01:02:40.790 align:middle line:84% So it is possible to come up in this class with ideas 01:02:40.790 --> 01:02:44.330 align:middle line:90% that are game changing. 01:02:44.330 --> 01:02:46.190 align:middle line:84% If you are taking the class for credit, 01:02:46.190 --> 01:02:50.690 align:middle line:84% you'll also take class notes for the lectures. 01:02:50.690 --> 01:02:55.520 align:middle line:84% We'll have plenty of guest talks in line and online discussion. 01:02:55.520 --> 01:02:58.040 align:middle line:90% 01:02:58.040 --> 01:03:01.460 align:middle line:84% This class, most of the materials is on slides. 01:03:01.460 --> 01:03:04.580 align:middle line:84% But starting next week, about half the material 01:03:04.580 --> 01:03:07.040 align:middle line:84% will be on slides, and the other half 01:03:07.040 --> 01:03:10.785 align:middle line:84% will be just on whiteboards and demos and so on. 01:03:10.785 --> 01:03:12.410 align:middle line:84% So if you're looking at a slide, you'll 01:03:12.410 --> 01:03:15.030 align:middle line:84% get an idea of the type of material we're covering, 01:03:15.030 --> 01:03:19.580 align:middle line:84% but what I realized last year was 01:03:19.580 --> 01:03:22.520 align:middle line:84% most of the discussion was actually not 01:03:22.520 --> 01:03:23.525 align:middle line:90% captured in the slides. 01:03:23.525 --> 01:03:26.480 align:middle line:90% 01:03:26.480 --> 01:03:28.970 align:middle line:90% And if you're a listener-- 01:03:28.970 --> 01:03:30.680 align:middle line:84% and I a lot of you are going to be 01:03:30.680 --> 01:03:33.900 align:middle line:84% listeners in this class, which is perfectly fine. 01:03:33.900 --> 01:03:38.540 align:middle line:84% But what I would like to do is participate in discussion. 01:03:38.540 --> 01:03:40.160 align:middle line:90% Bring in a new viewpoint. 01:03:40.160 --> 01:03:41.690 align:middle line:90% I want to learn from you. 01:03:41.690 --> 01:03:44.090 align:middle line:84% I'm glad Mathias is here, for example. 01:03:44.090 --> 01:03:46.550 align:middle line:84% He's the world expert world's expert on thermal imaging. 01:03:46.550 --> 01:03:50.990 align:middle line:84% So I want all of you to who are taking 01:03:50.990 --> 01:03:55.220 align:middle line:84% it, who are just listening in, to contribute to the discussion 01:03:55.220 --> 01:03:56.750 align:middle line:90% and give a new viewpoint. 01:03:56.750 --> 01:03:59.780 align:middle line:84% I will not be offended at all if you jump in 01:03:59.780 --> 01:04:03.200 align:middle line:84% and give a new perspective or a new reference 01:04:03.200 --> 01:04:06.620 align:middle line:90% about what we are discussing. 01:04:06.620 --> 01:04:12.020 align:middle line:84% And if you are a graduate student or a postdoc, 01:04:12.020 --> 01:04:15.080 align:middle line:84% then I would also like to spend some time 01:04:15.080 --> 01:04:21.860 align:middle line:84% in sharing with us one short presentation, maybe a cool idea 01:04:21.860 --> 01:04:24.440 align:middle line:90% or some new work you're doing. 01:04:24.440 --> 01:04:26.690 align:middle line:84% But that's what I would expect if you are not 01:04:26.690 --> 01:04:30.230 align:middle line:90% taking this class for credit. 01:04:30.230 --> 01:04:34.280 align:middle line:84% In terms of the credit breakdown, we have about four 01:04:34.280 --> 01:04:36.590 align:middle line:84% or three assignments depending on which version you 01:04:36.590 --> 01:04:41.900 align:middle line:84% are taking 40%, final project, 30%, midterm 40% and class 01:04:41.900 --> 01:04:42.590 align:middle line:90% participation. 01:04:42.590 --> 01:04:46.040 align:middle line:84% That includes discussions often online as well as 01:04:46.040 --> 01:04:48.710 align:middle line:90% taking the notes. 01:04:48.710 --> 01:04:51.490 align:middle line:90% Pre-requisites. 01:04:51.490 --> 01:04:52.850 align:middle line:90% Let's see if I have-- 01:04:52.850 --> 01:04:56.530 align:middle line:84% I'll come back to that, another slide. 01:04:56.530 --> 01:04:59.200 align:middle line:84% The emphasis for this class is really 01:04:59.200 --> 01:05:02.315 align:middle line:84% on fundamental techniques in imaging. 01:05:02.315 --> 01:05:03.190 align:middle line:90% Yeah, it's coming up. 01:05:03.190 --> 01:05:06.670 align:middle line:90% It's going to be fun. 01:05:06.670 --> 01:05:09.250 align:middle line:84% And in class as well as in homework, 01:05:09.250 --> 01:05:12.250 align:middle line:90% emphasis is on techniques. 01:05:12.250 --> 01:05:15.010 align:middle line:84% And they include all these key words, signal processing, 01:05:15.010 --> 01:05:18.220 align:middle line:84% applied optics graphics vision, online photo collection, 01:05:18.220 --> 01:05:21.680 align:middle line:84% statistical techniques, electronics, visual arts, 01:05:21.680 --> 01:05:22.180 align:middle line:90% and so on. 01:05:22.180 --> 01:05:24.830 align:middle line:84% And so in that sense it's not a discussion class. 01:05:24.830 --> 01:05:27.950 align:middle line:84% We're going to learn about techniques. 01:05:27.950 --> 01:05:32.840 align:middle line:84% All right, there is entertainment here. 01:05:32.840 --> 01:05:35.910 align:middle line:84% It's very addictive to play with cameras. 01:05:35.910 --> 01:05:36.600 align:middle line:90% OK. 01:05:36.600 --> 01:05:38.850 align:middle line:84% So if you can just focus here for a couple of minutes. 01:05:38.850 --> 01:05:41.340 align:middle line:90% 01:05:41.340 --> 01:05:45.540 align:middle line:84% And there are three areas we want to focus on. 01:05:45.540 --> 01:05:48.540 align:middle line:84% And just keep this in mind and know this. 01:05:48.540 --> 01:05:50.310 align:middle line:84% We're going to focus on photography. 01:05:50.310 --> 01:05:53.290 align:middle line:84% We're going to focus on active techniques, real time 01:05:53.290 --> 01:05:53.790 align:middle line:90% Techniques. 01:05:53.790 --> 01:05:56.310 align:middle line:84% And we're going to focus on scientific imaging. 01:05:56.310 --> 01:05:58.200 align:middle line:84% So within photography, we're going 01:05:58.200 --> 01:06:00.390 align:middle line:90% to think in higher dimensions. 01:06:00.390 --> 01:06:02.970 align:middle line:84% We're not going to think about just HDR 01:06:02.970 --> 01:06:04.720 align:middle line:90% or focus stack and so on. 01:06:04.720 --> 01:06:07.350 align:middle line:84% We're going to think about high dimensional imaging light 01:06:07.350 --> 01:06:12.630 align:middle line:84% fields, thermal imaging range cameras, and so on. 01:06:12.630 --> 01:06:15.120 align:middle line:84% And active computer vision, we're 01:06:15.120 --> 01:06:17.460 align:middle line:84% going to think about HCI applications, 01:06:17.460 --> 01:06:19.770 align:middle line:84% robotics, tracking and segmentation 01:06:19.770 --> 01:06:22.440 align:middle line:84% and how we can change the game by using 01:06:22.440 --> 01:06:24.540 align:middle line:90% feature revealing cameras. 01:06:24.540 --> 01:06:27.180 align:middle line:84% And a lot of concepts from scientific imaging 01:06:27.180 --> 01:06:29.400 align:middle line:84% such as composition sensing, wavefront 01:06:29.400 --> 01:06:32.220 align:middle line:84% coding, deconvolution, tomography, point square 01:06:32.220 --> 01:06:34.390 align:middle line:90% function, and so on-- 01:06:34.390 --> 01:06:38.520 align:middle line:84% And at first glance, these three areas might look very distinct 01:06:38.520 --> 01:06:41.520 align:middle line:84% and have very different techniques. 01:06:41.520 --> 01:06:45.150 align:middle line:84% But fortunately they all use very similar principles. 01:06:45.150 --> 01:06:47.800 align:middle line:84% And what we realized over the course 01:06:47.800 --> 01:06:51.990 align:middle line:84% is that this fusion of this dissimilar ideas-- 01:06:51.990 --> 01:06:54.420 align:middle line:84% I mean, tomography and wavefront coding 01:06:54.420 --> 01:06:57.780 align:middle line:84% seem very far away from traditional photography or HCI. 01:06:57.780 --> 01:07:00.390 align:middle line:84% But you'll realize that many of the problems you 01:07:00.390 --> 01:07:05.670 align:middle line:84% may be encountering and many of the visual arts 01:07:05.670 --> 01:07:10.320 align:middle line:84% that you might be interested in could be impacted from some 01:07:10.320 --> 01:07:14.470 align:middle line:90% of these new techniques. 01:07:14.470 --> 01:07:18.171 align:middle line:84% So in terms of the prerequisites. 01:07:18.171 --> 01:07:20.800 align:middle line:84% The last time I taught this course, 01:07:20.800 --> 01:07:24.180 align:middle line:84% there was a request to be supportive of students 01:07:24.180 --> 01:07:25.572 align:middle line:90% with different backgrounds. 01:07:25.572 --> 01:07:27.030 align:middle line:84% So this is what we're going to try. 01:07:27.030 --> 01:07:31.260 align:middle line:84% We're going to try to tracks, a software only tract 01:07:31.260 --> 01:07:34.110 align:middle line:90% and a software, hardware tract. 01:07:34.110 --> 01:07:36.990 align:middle line:84% And software intensive tract is for those of you 01:07:36.990 --> 01:07:39.870 align:middle line:84% who just because of your interests 01:07:39.870 --> 01:07:43.590 align:middle line:84% or want to do things in a particular way-- 01:07:43.590 --> 01:07:47.580 align:middle line:84% and you may be able to use some key based software. 01:07:47.580 --> 01:07:50.610 align:middle line:84% And the software hardware tract you will be using a lot 01:07:50.610 --> 01:07:56.490 align:middle line:84% of programming, OpenCV, MATLAB, C++ , Flash, whatever you like, 01:07:56.490 --> 01:07:57.210 align:middle line:90% Java. 01:07:57.210 --> 01:07:58.502 align:middle line:90% It doesn't really matter to me. 01:07:58.502 --> 01:08:01.290 align:middle line:84% But you'll be doing a lot of programming. 01:08:01.290 --> 01:08:03.750 align:middle line:84% And those of you who might be thinking about the software 01:08:03.750 --> 01:08:05.010 align:middle line:90% only track-- 01:08:05.010 --> 01:08:06.600 align:middle line:84% and you might be trying to use GUI. 01:08:06.600 --> 01:08:09.370 align:middle line:84% Actually it will be a lot of work also. 01:08:09.370 --> 01:08:11.490 align:middle line:84% And those of you who use Photoshop 01:08:11.490 --> 01:08:14.760 align:middle line:84% know very well that if you want to do simple edits to a photo, 01:08:14.760 --> 01:08:17.939 align:middle line:84% you take an hour, two hour, sometimes six hours. 01:08:17.939 --> 01:08:19.680 align:middle line:84% And you'll realize that sometimes writing 01:08:19.680 --> 01:08:23.700 align:middle line:84% a small program you can do that task much faster, 01:08:23.700 --> 01:08:26.080 align:middle line:90% but it's your cost. 01:08:26.080 --> 01:08:28.080 align:middle line:84% What is helpful but not absolutely 01:08:28.080 --> 01:08:31.800 align:middle line:84% necessary is some knowledge in linear algebra, 01:08:31.800 --> 01:08:34.470 align:middle line:84% signal processing, image processing. 01:08:34.470 --> 01:08:40.170 align:middle line:84% But what's critical is that you should be able to think in 3D. 01:08:40.170 --> 01:08:45.300 align:middle line:84% And this is a skill that's absolutely necessary. 01:08:45.300 --> 01:08:48.189 align:middle line:84% If you're going to think about in higher dimensions. 01:08:48.189 --> 01:08:50.189 align:middle line:84% 3D is just a beginning, but [? we're ?] thinking 01:08:50.189 --> 01:08:54.760 align:middle line:90% about 40, 60, 80 and so on. 01:08:54.760 --> 01:08:59.220 align:middle line:84% Now we're trying to keep the math to basic essentials. 01:08:59.220 --> 01:09:04.319 align:middle line:84% I like to use a lot of diagrams and visual analogies 01:09:04.319 --> 01:09:08.520 align:middle line:84% in the visual analogies to explain the concepts. 01:09:08.520 --> 01:09:10.470 align:middle line:84% But when you're doing your own assignments, 01:09:10.470 --> 01:09:12.779 align:middle line:84% you will have to go back to the questions. 01:09:12.779 --> 01:09:14.670 align:middle line:84% I might just flash the question, but I'll 01:09:14.670 --> 01:09:21.569 align:middle line:84% try to explain that again by drawing on the board. 01:09:21.569 --> 01:09:25.500 align:middle line:84% But it's possible for you to go through the whole class 01:09:25.500 --> 01:09:29.279 align:middle line:84% without actually writing down a lot of equations. 01:09:29.279 --> 01:09:33.120 align:middle line:84% At the same time, many of these concepts are complex, 01:09:33.120 --> 01:09:36.214 align:middle line:84% and they will arrive at a very fast pace. 01:09:36.214 --> 01:09:38.609 align:middle line:84% And we'll be discussing a lot of concepts. 01:09:38.609 --> 01:09:41.310 align:middle line:84% Now, if you're the kind of person who can just sit 01:09:41.310 --> 01:09:44.779 align:middle line:84% back and watch a presentation and grasp a concept, 01:09:44.779 --> 01:09:45.779 align:middle line:90% where you ask questions. 01:09:45.779 --> 01:09:47.160 align:middle line:90% It's interactive. 01:09:47.160 --> 01:09:49.492 align:middle line:90% This is an ideal class. 01:09:49.492 --> 01:09:51.450 align:middle line:84% On the other hand, if you're the kind of person 01:09:51.450 --> 01:09:52.950 align:middle line:84% who likes to really look at the math 01:09:52.950 --> 01:09:57.030 align:middle line:84% and see what the relationship between the variables 01:09:57.030 --> 01:09:59.950 align:middle line:84% is and so on, that's fine as well. 01:09:59.950 --> 01:10:01.770 align:middle line:84% So you need to have one of those two skills 01:10:01.770 --> 01:10:06.250 align:middle line:84% to be able to do well in this class. 01:10:06.250 --> 01:10:08.672 align:middle line:84% Now assignments for this class is also 01:10:08.672 --> 01:10:10.130 align:middle line:84% going to be a little bit different. 01:10:10.130 --> 01:10:12.070 align:middle line:84% As I said during the class, you'll 01:10:12.070 --> 01:10:15.100 align:middle line:84% be listening to a lot of advanced and complex concepts. 01:10:15.100 --> 01:10:18.220 align:middle line:84% But the assignments are structured in such a way 01:10:18.220 --> 01:10:21.410 align:middle line:84% that they have increasing level of sophistication. 01:10:21.410 --> 01:10:24.850 align:middle line:84% So you can do pretty well without much background 01:10:24.850 --> 01:10:27.730 align:middle line:84% for up to the first 60% or 70% of the assignment. 01:10:27.730 --> 01:10:33.220 align:middle line:84% But to do the last 10% or 20% you will need a good background 01:10:33.220 --> 01:10:34.515 align:middle line:90% in some of these areas. 01:10:34.515 --> 01:10:35.890 align:middle line:84% So that's another way of thinking 01:10:35.890 --> 01:10:39.500 align:middle line:84% about how you might be able to take this class, 01:10:39.500 --> 01:10:43.510 align:middle line:84% even if you don't have very strong mathematical, algebra, 01:10:43.510 --> 01:10:44.890 align:middle line:90% or signal processing background. 01:10:44.890 --> 01:10:48.010 align:middle line:84% So you can do pretty well, up to the 60% or 70% of. 01:10:48.010 --> 01:10:51.100 align:middle line:84% It and again in the superior of supporting students 01:10:51.100 --> 01:10:53.320 align:middle line:84% with varying background, what I will do 01:10:53.320 --> 01:10:56.680 align:middle line:84% is I will normalize your performance 01:10:56.680 --> 01:10:59.690 align:middle line:84% in those homeworks based on what background you have. 01:10:59.690 --> 01:11:03.490 align:middle line:84% So if I know that you don't have a linear algebra background, 01:11:03.490 --> 01:11:05.230 align:middle line:84% and you're taking this class, I will 01:11:05.230 --> 01:11:08.740 align:middle line:84% think that if you reach 70% of that assignment 01:11:08.740 --> 01:11:10.430 align:middle line:90% you have done a pretty good job. 01:11:10.430 --> 01:11:12.520 align:middle line:84% So I'll normalize it again by how much 01:11:12.520 --> 01:11:17.090 align:middle line:90% you know in that particular-- 01:11:17.090 --> 01:11:19.810 align:middle line:84% the knowledge required for that particular assignment. 01:11:19.810 --> 01:11:24.555 align:middle line:84% So you can really pick your level of how you want to do it. 01:11:24.555 --> 01:11:26.930 align:middle line:84% And we did this last time, and it worked out pretty well. 01:11:26.930 --> 01:11:30.200 align:middle line:84% So we'll see if we can do the same thing. 01:11:30.200 --> 01:11:32.830 align:middle line:84% And those of you who are taking the 131 01:11:32.830 --> 01:11:35.440 align:middle line:84% class for undergraduate, please come and talk to me. 01:11:35.440 --> 01:11:37.210 align:middle line:84% And we can similarly figure out based 01:11:37.210 --> 01:11:40.240 align:middle line:84% on the classes you have already taken, how we can structure 01:11:40.240 --> 01:11:41.230 align:middle line:90% those assignments. 01:11:41.230 --> 01:11:44.110 align:middle line:90% 01:11:44.110 --> 01:11:46.720 align:middle line:84% For all those assignments, the four or three assignments, 01:11:46.720 --> 01:11:48.190 align:middle line:90% there's always an option. 01:11:48.190 --> 01:11:50.930 align:middle line:84% There are two assignments, and you can pick any one of them. 01:11:50.930 --> 01:11:53.110 align:middle line:84% So that gives you, again, an ability 01:11:53.110 --> 01:11:57.100 align:middle line:84% to pick an assignment that's appropriate for your level 01:11:57.100 --> 01:12:00.130 align:middle line:90% of understanding. 01:12:00.130 --> 01:12:03.070 align:middle line:90% All right, so let's see. 01:12:03.070 --> 01:12:07.930 align:middle line:84% Any programming environment is fine, OnePlus notes. 01:12:07.930 --> 01:12:11.170 align:middle line:84% Send me an email, raskar@media.mit.edu, 01:12:11.170 --> 01:12:13.090 align:middle line:90% to put you on the mailing list. 01:12:13.090 --> 01:12:15.303 align:middle line:84% And we also have a sign up sheet that I'll 01:12:15.303 --> 01:12:16.345 align:middle line:90% pass around in the break. 01:12:16.345 --> 01:12:19.090 align:middle line:90% 01:12:19.090 --> 01:12:22.910 align:middle line:84% And, remember, our class runs into the happy hour on Friday. 01:12:22.910 --> 01:12:26.530 align:middle line:84% So after the class, we'll all go over to the marriage house 01:12:26.530 --> 01:12:32.200 align:middle line:84% and continue our discussion over beer, if you are over 18. 01:12:32.200 --> 01:12:33.730 align:middle line:90% No, 21. 01:12:33.730 --> 01:12:34.390 align:middle line:90% Is it 21? 01:12:34.390 --> 01:12:38.850 align:middle line:90% 01:12:38.850 --> 01:12:41.610 align:middle line:90% So that's at 4:30. 01:12:41.610 --> 01:12:47.850 align:middle line:84% This is a rough outline of how we're going to proceed. 01:12:47.850 --> 01:12:50.760 align:middle line:90% We finish on December 4th. 01:12:50.760 --> 01:12:54.690 align:middle line:84% It's funny because the classes end on December 10th. 01:12:54.690 --> 01:12:57.240 align:middle line:84% And our last class would have been December 11th. 01:12:57.240 --> 01:13:01.770 align:middle line:84% So I think having a Friday class with no final exam 01:13:01.770 --> 01:13:04.410 align:middle line:84% means you finish really, really early. 01:13:04.410 --> 01:13:06.120 align:middle line:84% So the semester ends on December 4th, 01:13:06.120 --> 01:13:07.350 align:middle line:90% which means final projects. 01:13:07.350 --> 01:13:10.410 align:middle line:84% Unfortunately, the week before that is thanksgiving. 01:13:10.410 --> 01:13:12.540 align:middle line:84% So many of you will be thanking me 01:13:12.540 --> 01:13:17.700 align:middle line:84% for the delay and the procrastination that 01:13:17.700 --> 01:13:21.340 align:middle line:84% led you to work through the Thanksgiving break. 01:13:21.340 --> 01:13:23.170 align:middle line:90% Yeah, there's just a list here. 01:13:23.170 --> 01:13:23.670 align:middle line:90% All right. 01:13:23.670 --> 01:13:30.270 align:middle line:84% So one of the things we will not cover or cover not in detail 01:13:30.270 --> 01:13:37.500 align:middle line:84% are the art and aesthetics of photography, the 4343. 01:13:37.500 --> 01:13:40.320 align:middle line:84% Software image manipulation, there's 01:13:40.320 --> 01:13:44.400 align:middle line:84% a great course by Fredo Durand on digital computational 01:13:44.400 --> 01:13:47.020 align:middle line:84% photography which is completely focused on software. 01:13:47.020 --> 01:13:50.040 align:middle line:84% So these two classes, first class and my class, 01:13:50.040 --> 01:13:52.620 align:middle line:84% is actually a very good complement of each other 01:13:52.620 --> 01:13:54.750 align:middle line:84% because in this class, the emphasis is more 01:13:54.750 --> 01:13:57.480 align:middle line:84% on hardware and optics and sensors 01:13:57.480 --> 01:13:58.860 align:middle line:90% and of course, software. 01:13:58.860 --> 01:14:02.190 align:middle line:84% But emphasis is on the hands on elements. 01:14:02.190 --> 01:14:07.230 align:middle line:84% And I believe Fredo's not going to teach the class in spring. 01:14:07.230 --> 01:14:10.440 align:middle line:84% But Will Freeman, who's also a great instructor, 01:14:10.440 --> 01:14:13.098 align:middle line:84% is going to teach the class in spring. 01:14:13.098 --> 01:14:13.890 align:middle line:90% That's what I hear. 01:14:13.890 --> 01:14:16.650 align:middle line:90% I haven't confirmed it. 01:14:16.650 --> 01:14:19.660 align:middle line:84% And there are excellent classes in computer vision. 01:14:19.660 --> 01:14:23.040 align:middle line:84% I think, Ted, you are teaching a class on scene perception. 01:14:23.040 --> 01:14:23.592 align:middle line:90% TED: Shape. 01:14:23.592 --> 01:14:25.050 align:middle line:84% RAMESH RASKAR: For shape perception 01:14:25.050 --> 01:14:28.425 align:middle line:90% that's on Mondays, just Mondays. 01:14:28.425 --> 01:14:29.592 align:middle line:90% TED: No, we don't know when. 01:14:29.592 --> 01:14:30.633 align:middle line:90% RAMESH RASKAR: All right. 01:14:30.633 --> 01:14:32.550 align:middle line:84% TED: First class is Monday, [INAUDIBLE].. 01:14:32.550 --> 01:14:35.700 align:middle line:84% RAMESH RASKAR: OK, but I already sent an email about Ted's class 01:14:35.700 --> 01:14:37.060 align:middle line:90% on the mailing list. 01:14:37.060 --> 01:14:39.990 align:middle line:84% So once I hear from you, I'll send it out again. 01:14:39.990 --> 01:14:42.570 align:middle line:84% There are a couple of great optics classes. 01:14:42.570 --> 01:14:44.160 align:middle line:90% This class is not about optics. 01:14:44.160 --> 01:14:45.660 align:middle line:84% You'll be learning a lot of concept. 01:14:45.660 --> 01:14:47.640 align:middle line:84% But the emphasis is not on optics. 01:14:47.640 --> 01:14:51.030 align:middle line:84% The emphasis is on imaging and photography. 01:14:51.030 --> 01:14:52.620 align:middle line:84% We will not learn about Photoshop. 01:14:52.620 --> 01:14:55.218 align:middle line:84% Actually, I don't even know Photoshop that very well. 01:14:55.218 --> 01:14:57.510 align:middle line:84% So if you're going to do your assignments in Photoshop, 01:14:57.510 --> 01:14:58.770 align:middle line:90% please come and tell me. 01:14:58.770 --> 01:15:01.470 align:middle line:90% I'll look forward to it. 01:15:01.470 --> 01:15:03.240 align:middle line:84% And we won't talk about anything that's 01:15:03.240 --> 01:15:05.550 align:middle line:84% included in the instruction manual of the camera. 01:15:05.550 --> 01:15:08.490 align:middle line:84% So you will not learn about how to set the exposure 01:15:08.490 --> 01:15:10.020 align:middle line:90% and how to change the aperture. 01:15:10.020 --> 01:15:14.010 align:middle line:84% I'm happy to do a separate crash course, one of the evenings. 01:15:14.010 --> 01:15:19.230 align:middle line:84% And we can all sit together and do a course. 01:15:19.230 --> 01:15:21.510 align:middle line:84% On the many people here it is super expert 01:15:21.510 --> 01:15:24.180 align:middle line:84% with all kinds of cameras, and he'll 01:15:24.180 --> 01:15:25.360 align:middle line:90% be happy to do that as well. 01:15:25.360 --> 01:15:27.360 align:middle line:84% So there are a lot of resources available if you 01:15:27.360 --> 01:15:28.320 align:middle line:90% have those questions. 01:15:28.320 --> 01:15:31.960 align:middle line:84% But this class is not really a program. 01:15:31.960 --> 01:15:35.380 align:middle line:84% So as I said a few classes I teach a class 01:15:35.380 --> 01:15:38.950 align:middle line:84% that's more discussion oriented in spring, 01:15:38.950 --> 01:15:42.820 align:middle line:84% the computational photography class and optics class. 01:15:42.820 --> 01:15:47.980 align:middle line:84% And Professor Han, I don't if he's teaching it in Spring. 01:15:47.980 --> 01:15:50.290 align:middle line:90% Yeah, I don't know. 01:15:50.290 --> 01:15:52.870 align:middle line:84% All right, so any questions about the structure 01:15:52.870 --> 01:15:57.800 align:middle line:84% of the class and what you expect? 01:15:57.800 --> 01:15:58.315 align:middle line:90% Rose, yes. 01:15:58.315 --> 01:16:01.390 align:middle line:84% AUDIENCE: Will you be posting the assignments in advance 01:16:01.390 --> 01:16:03.270 align:middle line:84% so we can kind of see what sort of work 01:16:03.270 --> 01:16:05.583 align:middle line:84% will we be doing and engaging with during class? 01:16:05.583 --> 01:16:07.000 align:middle line:84% RAMESH RASKAR: Excellent question. 01:16:07.000 --> 01:16:11.110 align:middle line:84% The question was, will I be posting assignments in advance? 01:16:11.110 --> 01:16:13.930 align:middle line:84% What's going to happen is based on the feedback I receive. 01:16:13.930 --> 01:16:15.530 align:middle line:84% Some of the assignments will change. 01:16:15.530 --> 01:16:23.420 align:middle line:84% But if you look at the OCW page from last year for this course, 01:16:23.420 --> 01:16:25.450 align:middle line:84% which is only an indication by the way 01:16:25.450 --> 01:16:27.980 align:middle line:84% because of course it's changing significantly this year, 01:16:27.980 --> 01:16:29.800 align:middle line:90% you will see the-- 01:16:29.800 --> 01:16:30.760 align:middle line:90% let's see the projects. 01:16:30.760 --> 01:16:36.350 align:middle line:90% 01:16:36.350 --> 01:16:38.300 align:middle line:84% You'll see the type of assignments 01:16:38.300 --> 01:16:39.720 align:middle line:90% that were given out. 01:16:39.720 --> 01:16:41.870 align:middle line:90% It has all the details as well. 01:16:41.870 --> 01:16:46.020 align:middle line:84% So four assignments, writing and so on. 01:16:46.020 --> 01:16:48.650 align:middle line:84% So you can already get a sense of the type of assignments 01:16:48.650 --> 01:16:49.560 align:middle line:90% that were given. 01:16:49.560 --> 01:16:55.200 align:middle line:90% And so, again, the options. 01:16:55.200 --> 01:16:57.420 align:middle line:90% There is 4A and 4B for example. 01:16:57.420 --> 01:17:01.410 align:middle line:84% So you can choose which one you would like to do. 01:17:01.410 --> 01:17:05.280 align:middle line:84% Again, the OpenCourseWare page is for last year's course. 01:17:05.280 --> 01:17:07.410 align:middle line:84% And we are recording it for this year. 01:17:07.410 --> 01:17:10.990 align:middle line:84% But it's not up here till next fall. 01:17:10.990 --> 01:17:14.760 align:middle line:84% So there's a long delay before the model appears. 01:17:14.760 --> 01:17:18.220 align:middle line:84% And next year we'll be doing something entirely different 01:17:18.220 --> 01:17:18.720 align:middle line:90% again. 01:17:18.720 --> 01:17:22.800 align:middle line:90% 01:17:22.800 --> 01:17:23.610 align:middle line:90% Other questions? 01:17:23.610 --> 01:17:28.470 align:middle line:90% 01:17:28.470 --> 01:17:30.890 align:middle line:84% All right, so let's go around very briefly. 01:17:30.890 --> 01:17:33.350 align:middle line:84% I realize there are quite a few people. 01:17:33.350 --> 01:17:37.430 align:middle line:84% So maybe 30 seconds per person no more than 01:17:37.430 --> 01:17:41.510 align:middle line:84% that just to get a sense of who's here 01:17:41.510 --> 01:17:43.800 align:middle line:90% and why you are here. 01:17:43.800 --> 01:17:46.440 align:middle line:90% So before we get started. 01:17:46.440 --> 01:17:48.090 align:middle line:90% Are you a photographer? 01:17:48.090 --> 01:17:51.310 align:middle line:90% 01:17:51.310 --> 01:17:52.580 align:middle line:90% All right, almost everybody. 01:17:52.580 --> 01:17:54.340 align:middle line:84% How many of you are videographer? 01:17:54.340 --> 01:17:55.135 align:middle line:90% You create videos. 01:17:55.135 --> 01:17:58.280 align:middle line:90% 01:17:58.280 --> 01:17:58.880 align:middle line:90% It's funny. 01:17:58.880 --> 01:18:00.838 align:middle line:84% Why there's a distinction between photographers 01:18:00.838 --> 01:18:02.990 align:middle line:90% and videographers? 01:18:02.990 --> 01:18:08.250 align:middle line:84% Do you use cameras for computer vision. 01:18:08.250 --> 01:18:09.610 align:middle line:90% Pretty good. 01:18:09.610 --> 01:18:12.530 align:middle line:84% Do you use cameras for real time processing 01:18:12.530 --> 01:18:16.170 align:middle line:90% like HCI or robotics and so on? 01:18:16.170 --> 01:18:17.940 align:middle line:90% OK. 01:18:17.940 --> 01:18:21.770 align:middle line:84% Do you have background in optics or sensors? 01:18:21.770 --> 01:18:22.740 align:middle line:90% OK, all right. 01:18:22.740 --> 01:18:25.030 align:middle line:84% So we have pretty good distribution here. 01:18:25.030 --> 01:18:27.330 align:middle line:84% So so let's start and very quickly go 01:18:27.330 --> 01:18:33.090 align:middle line:84% to the just your name, department here, 01:18:33.090 --> 01:18:35.430 align:middle line:84% and why you're here, just 30 seconds. 01:18:35.430 --> 01:18:41.440 align:middle line:90% 01:18:41.440 --> 01:18:44.770 align:middle line:84% I take this opportunity to say that Professor Oliveira is 01:18:44.770 --> 01:18:47.320 align:middle line:84% the top scientist in graphics and vision from Brazil, 01:18:47.320 --> 01:18:48.700 align:middle line:90% and he'll be in our group. 01:18:48.700 --> 01:18:52.570 align:middle line:84% And we are sending him and Professor Mukaigawa 01:18:52.570 --> 01:18:56.530 align:middle line:84% who is a famous researcher from Osaka University 01:18:56.530 --> 01:18:59.560 align:middle line:84% and both of them, as well as Ankit Mohan, who 01:18:59.560 --> 01:19:02.680 align:middle line:84% is a scientist, those 3 and some additional people 01:19:02.680 --> 01:19:05.300 align:middle line:84% will be the mentors for this class. 01:19:05.300 --> 01:19:07.450 align:middle line:84% So if you have any questions, you can come to me 01:19:07.450 --> 01:19:11.290 align:middle line:84% or if you want to brainstorm about projects or ideas 01:19:11.290 --> 01:19:13.930 align:middle line:84% or have questions, they will be available as well 01:19:13.930 --> 01:19:15.270 align:middle line:90% during the week. 01:19:15.270 --> 01:19:21.300 align:middle line:90% 01:19:21.300 --> 01:19:23.100 align:middle line:84% Oh, this is amazing, just an amazing set 01:19:23.100 --> 01:19:29.845 align:middle line:84% of people, chemistry, arts, communication, vision, HCI. 01:19:29.845 --> 01:19:30.720 align:middle line:90% It's going to be fun. 01:19:30.720 --> 01:19:33.150 align:middle line:90% Night vision goggles. 01:19:33.150 --> 01:19:34.272 align:middle line:90% Night vision goggles. 01:19:34.272 --> 01:19:35.730 align:middle line:84% That would have taken us over 100k. 01:19:35.730 --> 01:19:38.830 align:middle line:90% 01:19:38.830 --> 01:19:39.330 align:middle line:90% All right. 01:19:39.330 --> 01:19:42.330 align:middle line:84% So let me just quickly make a couple of announcements, 01:19:42.330 --> 01:19:45.030 align:middle line:90% and we'll break. 01:19:45.030 --> 01:19:48.240 align:middle line:84% So there's a great opportunity for those of you 01:19:48.240 --> 01:19:49.740 align:middle line:84% who are taking this class or credit. 01:19:49.740 --> 01:19:53.490 align:middle line:84% And there's a conference on computational photography 01:19:53.490 --> 01:19:57.120 align:middle line:84% which is the second edition of Fredo Durand who some of you 01:19:57.120 --> 01:19:59.070 align:middle line:90% may know from CSAIL. 01:19:59.070 --> 01:20:03.810 align:middle line:84% And Mark LeVoy and Eric Zaleski from Microsoft 01:20:03.810 --> 01:20:07.140 align:middle line:84% did the first one, which was this year in April. 01:20:07.140 --> 01:20:11.610 align:middle line:84% And myself and [INAUDIBLE] from Toronto and Raphael Bastion 01:20:11.610 --> 01:20:14.130 align:middle line:84% from Colorado was one of the world's experts 01:20:14.130 --> 01:20:16.140 align:middle line:90% in wavefront coding. 01:20:16.140 --> 01:20:20.860 align:middle line:84% Three of us are doing the second edition of this conference. 01:20:20.860 --> 01:20:24.360 align:middle line:84% And it will be held here in the new building 01:20:24.360 --> 01:20:27.990 align:middle line:90% in the last week of March. 01:20:27.990 --> 01:20:31.300 align:middle line:84% And the papers are due in November. 01:20:31.300 --> 01:20:33.160 align:middle line:84% So if you're doing well in this class, 01:20:33.160 --> 01:20:36.300 align:middle line:84% and one of the project ideas is interesting, 01:20:36.300 --> 01:20:39.960 align:middle line:84% you could even attempt to submit a paper 01:20:39.960 --> 01:20:42.520 align:middle line:84% for this particular conference or at least 01:20:42.520 --> 01:20:44.710 align:middle line:84% as a demo or a poster or something like that. 01:20:44.710 --> 01:20:47.070 align:middle line:90% So it's right here on campus. 01:20:47.070 --> 01:20:50.130 align:middle line:84% And most likely we'll have additional opportunities 01:20:50.130 --> 01:20:52.910 align:middle line:84% who are not in the peer review track 01:20:52.910 --> 01:20:54.820 align:middle line:90% to show off your work as well. 01:20:54.820 --> 01:20:57.000 align:middle line:90% So this is really catching on. 01:20:57.000 --> 01:21:00.720 align:middle line:84% A lot of big companies like Nokia and Samsung 01:21:00.720 --> 01:21:05.970 align:middle line:84% and Canon and HP all have started big research groups 01:21:05.970 --> 01:21:08.670 align:middle line:84% in computational photography and computational cameras. 01:21:08.670 --> 01:21:13.410 align:middle line:90% So there's a lot of interest. 01:21:13.410 --> 01:21:17.610 align:middle line:84% Another thing we're going to do in this class is-- 01:21:17.610 --> 01:21:19.380 align:middle line:84% as I said, it is possible to come up 01:21:19.380 --> 01:21:21.280 align:middle line:84% with completely new ideas in this field. 01:21:21.280 --> 01:21:23.865 align:middle line:84% It's such a new field because of the intersection 01:21:23.865 --> 01:21:27.070 align:middle line:90% of lots of interesting domains. 01:21:27.070 --> 01:21:30.570 align:middle line:84% So we're going to learn how to come up with new ideas. 01:21:30.570 --> 01:21:33.750 align:middle line:84% And we're going to learn how to write a good paper. 01:21:33.750 --> 01:21:36.330 align:middle line:84% We're not going to do it exactly it in the class. 01:21:36.330 --> 01:21:39.120 align:middle line:84% But toward the semester during discussions 01:21:39.120 --> 01:21:43.200 align:middle line:84% with me and your mentors, we will help you actually 01:21:43.200 --> 01:21:46.110 align:middle line:90% write a good quality paper. 01:21:46.110 --> 01:21:48.420 align:middle line:84% And writing a good quality paper actually 01:21:48.420 --> 01:21:52.060 align:middle line:84% has really simple methodologies which is amazing. 01:21:52.060 --> 01:21:54.370 align:middle line:84% We never learn about it in a formal class. 01:21:54.370 --> 01:21:57.420 align:middle line:84% So we'll try to do that in this class 01:21:57.420 --> 01:21:59.790 align:middle line:84% because the final project is really important. 01:21:59.790 --> 01:22:03.690 align:middle line:84% So just deciding if an idea is worth pursuing 01:22:03.690 --> 01:22:05.110 align:middle line:90% is half the battle. 01:22:05.110 --> 01:22:08.130 align:middle line:84% And we'll help you do that, you can actually just use 01:22:08.130 --> 01:22:11.760 align:middle line:84% the Heilmeier's rules which the military uses 01:22:11.760 --> 01:22:14.205 align:middle line:84% to decide whether they should pursue a project. 01:22:14.205 --> 01:22:15.330 align:middle line:90% And raise simple questions. 01:22:15.330 --> 01:22:17.790 align:middle line:84% If we answer those questions, you can very quickly 01:22:17.790 --> 01:22:21.210 align:middle line:84% make a decision whether you should pursue that project. 01:22:21.210 --> 01:22:24.180 align:middle line:84% And as I said last year was extremely, extremely beautiful. 01:22:24.180 --> 01:22:29.970 align:middle line:84% There were papers that became SIGGRAPH submissions. 01:22:29.970 --> 01:22:32.040 align:middle line:84% Matt Hirsch, who took this class last year, 01:22:32.040 --> 01:22:35.760 align:middle line:84% went to SIGGRAPH and also in the student research competition 01:22:35.760 --> 01:22:38.400 align:middle line:84% this year based on the last project he did. 01:22:38.400 --> 01:22:40.890 align:middle line:84% And there are two major research teams 01:22:40.890 --> 01:22:43.652 align:middle line:84% that are coming out one in mechanical engineering. 01:22:43.652 --> 01:22:46.110 align:middle line:84% Some students, they're starting this multi hundred thousand 01:22:46.110 --> 01:22:48.880 align:middle line:84% dollar project based on the class project here and so on. 01:22:48.880 --> 01:22:52.170 align:middle line:84% So we really want you to focus on novel ideas that 01:22:52.170 --> 01:22:55.680 align:middle line:90% are cool and publishable. 01:22:55.680 --> 01:22:58.170 align:middle line:84% Of course, those of you who have a design background or art 01:22:58.170 --> 01:23:01.530 align:middle line:84% background, we also will try to think about how they can 01:23:01.530 --> 01:23:05.168 align:middle line:90% be given the right exposure. 01:23:05.168 --> 01:23:06.960 align:middle line:84% And when it comes to technical publication, 01:23:06.960 --> 01:23:08.670 align:middle line:84% there are some simple rules that you 01:23:08.670 --> 01:23:11.820 align:middle line:84% can follow to write a reasonable quality paper. 01:23:11.820 --> 01:23:14.410 align:middle line:90% So will help you do that. 01:23:14.410 --> 01:23:18.090 align:middle line:84% So let me stop there with this image 01:23:18.090 --> 01:23:20.160 align:middle line:84% to make you go a little bit dizzy. 01:23:20.160 --> 01:23:23.550 align:middle line:84% And then we'll reconvene in about 10 minutes. 01:23:23.550 --> 01:23:27.030 align:middle line:84% And in the second half, we will have a fast forward preview 01:23:27.030 --> 01:23:28.690 align:middle line:90% of the whole class. 01:23:28.690 --> 01:23:32.010 align:middle line:84% So we'll spend about 2 to 3 minutes 01:23:32.010 --> 01:23:33.360 align:middle line:90% on each of the projects. 01:23:33.360 --> 01:23:35.340 align:middle line:90% We have 12 classes. 01:23:35.340 --> 01:23:37.230 align:middle line:84% We'll spend about 5 minutes on each class, 01:23:37.230 --> 01:23:39.820 align:middle line:90% and we will see all the pieces. 01:23:39.820 --> 01:23:42.780 align:middle line:84% So in the break, we'll also have the IR. 01:23:42.780 --> 01:23:45.450 align:middle line:90% camera you can try that. 01:23:45.450 --> 01:23:53.280 align:middle line:84% And Dan [? Sarkis ?] suggested for all of us to build this-- 01:23:53.280 --> 01:23:55.000 align:middle line:90% what do you call it? 01:23:55.000 --> 01:23:58.507 align:middle line:84% A pinhole camera from just a piece of paper. 01:23:58.507 --> 01:24:00.090 align:middle line:84% So you can just take a piece of paper, 01:24:00.090 --> 01:24:02.920 align:middle line:84% cut it up, and build a pinhole camera. 01:24:02.920 --> 01:24:06.578 align:middle line:84% So if you want to do some projects in that space-- 01:24:06.578 --> 01:24:07.120 align:middle line:90% AUDIENCE: Hi. 01:24:07.120 --> 01:24:09.680 align:middle line:90% [SIDE CONVERSATION]