WEBVTT 00:00:00.000 --> 00:00:02.490 align:middle line:84% The following content is provided under a Creative 00:00:02.490 --> 00:00:03.730 align:middle line:90% Commons license. 00:00:03.730 --> 00:00:06.120 align:middle line:84% Your support will help MIT OpenCourseWare 00:00:06.120 --> 00:00:10.090 align:middle line:84% continue to offer high-quality educational resources for free. 00:00:10.090 --> 00:00:12.720 align:middle line:84% To make a donation or to view additional materials 00:00:12.720 --> 00:00:16.650 align:middle line:84% from hundreds of MIT courses, visit MIT OpenCourseWare 00:00:16.650 --> 00:00:19.370 align:middle line:90% at ocw.mit.edu. 00:00:19.370 --> 00:00:21.870 align:middle line:84% ROARKE HORSTMEYER: I'm a new student with the Camera Culture 00:00:21.870 --> 00:00:24.270 align:middle line:90% Group. 00:00:24.270 --> 00:00:26.730 align:middle line:84% But for a little over a year I worked 00:00:26.730 --> 00:00:29.430 align:middle line:84% at a company called Miter which, one of them's up the road. 00:00:29.430 --> 00:00:32.009 align:middle line:90% The other one's in DC. 00:00:32.009 --> 00:00:37.890 align:middle line:84% And we on and off worked on this project, myself 00:00:37.890 --> 00:00:39.480 align:middle line:84% with a guy named Gary [INAUDIBLE] 00:00:39.480 --> 00:00:42.750 align:middle line:84% and then Mark Levoy who is from Stanford. 00:00:42.750 --> 00:00:47.430 align:middle line:84% And together, we eventually put together a nice project. 00:00:47.430 --> 00:00:48.400 align:middle line:90% But it took a bit. 00:00:48.400 --> 00:00:53.580 align:middle line:84% So 2009, 2008, is kind of a general time frame we did this. 00:00:53.580 --> 00:00:55.170 align:middle line:84% But this is going along the lines 00:00:55.170 --> 00:01:01.350 align:middle line:84% of capturing multi-domain or multi-dimensional data 00:01:01.350 --> 00:01:04.349 align:middle line:84% spectral data or polarimetric information 00:01:04.349 --> 00:01:07.860 align:middle line:84% or high dynamic range with your camera 00:01:07.860 --> 00:01:10.420 align:middle line:84% but doing it in a single snapshot. 00:01:10.420 --> 00:01:13.810 align:middle line:84% So we'll discuss that in a little bit of detail. 00:01:13.810 --> 00:01:17.370 align:middle line:90% But first, just some background. 00:01:17.370 --> 00:01:21.990 align:middle line:84% Just giving you an idea of what I mean when I say multi-domain. 00:01:21.990 --> 00:01:23.820 align:middle line:84% So we know there's a ton of pixels. 00:01:23.820 --> 00:01:26.160 align:middle line:84% Ramesh showed some slides of the increase 00:01:26.160 --> 00:01:29.460 align:middle line:84% in number of pixels per sensor and whatnot. 00:01:29.460 --> 00:01:34.830 align:middle line:84% And the thrust of this research was finding a new way 00:01:34.830 --> 00:01:38.520 align:middle line:84% to use those pixels besides just spatial resolution. 00:01:38.520 --> 00:01:41.980 align:middle line:84% We think there's plenty of spatial resolution out there. 00:01:41.980 --> 00:01:47.160 align:middle line:84% What about color or some other interesting properties of light 00:01:47.160 --> 00:01:48.060 align:middle line:90% to capture it? 00:01:48.060 --> 00:01:50.250 align:middle line:84% So some of those include polarization. 00:01:50.250 --> 00:01:55.590 align:middle line:84% So here a group used a polar emitter to distinguish 00:01:55.590 --> 00:01:56.610 align:middle line:90% a man-made object-- 00:01:56.610 --> 00:02:01.470 align:middle line:84% this is metal-- from some plants. 00:02:01.470 --> 00:02:03.828 align:middle line:84% The way it works is metal reflects more polarized light 00:02:03.828 --> 00:02:06.120 align:middle line:84% than a plant does, which actually reflects a little bit 00:02:06.120 --> 00:02:08.160 align:middle line:90% of polarized light. 00:02:08.160 --> 00:02:10.560 align:middle line:84% And they pick out objects like that. 00:02:10.560 --> 00:02:13.590 align:middle line:84% I'll show some examples some of that what we did later, 00:02:13.590 --> 00:02:14.610 align:middle line:90% I think. 00:02:14.610 --> 00:02:16.110 align:middle line:84% Anyways, so you can see cool effects 00:02:16.110 --> 00:02:19.830 align:middle line:84% like [INAUDIBLE] in glasses or transparent media. 00:02:19.830 --> 00:02:21.450 align:middle line:84% And then, obviously, what's known 00:02:21.450 --> 00:02:22.860 align:middle line:90% for photographers is it helps-- 00:02:22.860 --> 00:02:26.340 align:middle line:90% 00:02:26.340 --> 00:02:34.350 align:middle line:84% I forget the word- but clarify haziness in images of water, 00:02:34.350 --> 00:02:37.260 align:middle line:84% because water reflects polarized light and so does the sky. 00:02:37.260 --> 00:02:40.650 align:middle line:84% So you get a clear blue sky with a polarizing filter 00:02:40.650 --> 00:02:42.390 align:middle line:90% for conventional photography. 00:02:42.390 --> 00:02:45.580 align:middle line:84% You can also do multispectral imaging, 00:02:45.580 --> 00:02:48.870 align:middle line:84% which is just capturing a very specific spectral information 00:02:48.870 --> 00:02:51.190 align:middle line:84% instead of just red, green, and blue, 00:02:51.190 --> 00:02:55.590 align:middle line:84% which are huge bands in the frequency dimension. 00:02:55.590 --> 00:02:59.280 align:middle line:84% These are very narrow-band images. 00:02:59.280 --> 00:03:01.120 align:middle line:84% These were both captured over time. 00:03:01.120 --> 00:03:03.180 align:middle line:90% So this one shows a aerial view. 00:03:03.180 --> 00:03:06.600 align:middle line:84% They're often used in airplanes to capture information 00:03:06.600 --> 00:03:08.260 align:middle line:90% about foliage or stuff. 00:03:08.260 --> 00:03:10.410 align:middle line:90% USGS is using them a lot. 00:03:10.410 --> 00:03:13.420 align:middle line:84% And then this is an example of detecting blood oxygenation. 00:03:13.420 --> 00:03:16.470 align:middle line:84% So this is an image of a thumb next to another thumb. 00:03:16.470 --> 00:03:18.310 align:middle line:84% One has a rubber band wrapped around it. 00:03:18.310 --> 00:03:19.270 align:middle line:90% The other one doesn't. 00:03:19.270 --> 00:03:21.400 align:middle line:84% So you can obviously tell the difference. 00:03:21.400 --> 00:03:22.900 align:middle line:84% And then high dynamic range imaging, 00:03:22.900 --> 00:03:25.210 align:middle line:90% which I think we talked about. 00:03:25.210 --> 00:03:28.062 align:middle line:84% So we're going to capture all that in a single snapshot. 00:03:28.062 --> 00:03:29.520 align:middle line:84% And basically what happens when you 00:03:29.520 --> 00:03:30.978 align:middle line:84% want to do that is you're presented 00:03:30.978 --> 00:03:33.930 align:middle line:84% with this huge dimensionality mismatch, which 00:03:33.930 --> 00:03:39.360 align:middle line:84% is true of all camera capture techniques. 00:03:39.360 --> 00:03:42.270 align:middle line:84% You just have this two dimensional sensor. 00:03:42.270 --> 00:03:44.460 align:middle line:90% And it's small and it's flat. 00:03:44.460 --> 00:03:45.945 align:middle line:90% Soon, it might not be flat. 00:03:45.945 --> 00:03:46.817 align:middle line:90% But right now it is. 00:03:46.817 --> 00:03:48.900 align:middle line:84% And you also have the time dimension to work over. 00:03:48.900 --> 00:03:51.000 align:middle line:84% But you have these seven or more, 00:03:51.000 --> 00:03:53.550 align:middle line:84% arguably, seven dimensions of information out there. 00:03:53.550 --> 00:03:55.800 align:middle line:84% You have a four-dimensional light field, 00:03:55.800 --> 00:03:57.570 align:middle line:84% which I guess you'll learn about more. 00:03:57.570 --> 00:03:58.778 align:middle line:90% Some of you might know about. 00:03:58.778 --> 00:04:01.740 align:middle line:84% He also has a temporal, the changing of whatever scene 00:04:01.740 --> 00:04:02.490 align:middle line:90% you're looking at. 00:04:02.490 --> 00:04:05.830 align:middle line:84% And you have spectral polarization information. 00:04:05.830 --> 00:04:07.470 align:middle line:84% So there's a lot of different ways 00:04:07.470 --> 00:04:10.050 align:middle line:84% that people have come up with to capture this. 00:04:10.050 --> 00:04:11.970 align:middle line:90% And I'll talk about some. 00:04:11.970 --> 00:04:15.220 align:middle line:84% One of the most direct is just doing it over time. 00:04:15.220 --> 00:04:18.450 align:middle line:84% So this is a picture of one of the first color motion picture 00:04:18.450 --> 00:04:19.120 align:middle line:90% processes. 00:04:19.120 --> 00:04:20.829 align:middle line:90% It's called Kinema color. 00:04:20.829 --> 00:04:23.640 align:middle line:84% What they did is they just put a rotating green and red filter 00:04:23.640 --> 00:04:28.980 align:middle line:84% in front of a regular old movie film camera. 00:04:28.980 --> 00:04:30.660 align:middle line:84% And it captured sequential images. 00:04:30.660 --> 00:04:32.100 align:middle line:90% Then when they projected it-- 00:04:32.100 --> 00:04:33.840 align:middle line:84% it didn't have blue for some reason. 00:04:33.840 --> 00:04:35.040 align:middle line:90% I don't know why. 00:04:35.040 --> 00:04:37.710 align:middle line:84% But projected it, they just projected it 00:04:37.710 --> 00:04:39.690 align:middle line:84% through the same rotating color filter. 00:04:39.690 --> 00:04:42.470 align:middle line:84% And your eyes couldn't distinguish the fast frame 00:04:42.470 --> 00:04:42.970 align:middle line:90% rate. 00:04:42.970 --> 00:04:44.970 align:middle line:90% So it looked like a color code. 00:04:44.970 --> 00:04:46.720 align:middle line:84% And then some of the things have been done 00:04:46.720 --> 00:04:49.190 align:middle line:90% with hyperspectral imaging. 00:04:49.190 --> 00:04:51.370 align:middle line:84% So the way I push through hyperspectral imager works 00:04:51.370 --> 00:04:53.140 align:middle line:84% is it's a two-dimensional sensor. 00:04:53.140 --> 00:04:55.760 align:middle line:84% One dimension is used to capture spatial information. 00:04:55.760 --> 00:04:58.160 align:middle line:84% And the other is used to capture spectral information. 00:04:58.160 --> 00:05:00.940 align:middle line:84% And you put it up on a plane, and it flies. 00:05:00.940 --> 00:05:03.200 align:middle line:84% And a sweeps out the second spatial dimension. 00:05:03.200 --> 00:05:06.060 align:middle line:84% So you're capturing this three-dimensional data cube 00:05:06.060 --> 00:05:07.545 align:middle line:90% and over time, two dimensions. 00:05:07.545 --> 00:05:11.020 align:middle line:84% But you're capturing spectrum [INAUDIBLE].. 00:05:11.020 --> 00:05:13.660 align:middle line:90% It's a little bit different. 00:05:13.660 --> 00:05:16.390 align:middle line:84% And then this is very similar to what RK was talking about. 00:05:16.390 --> 00:05:17.807 align:middle line:84% But there's tons of different ways 00:05:17.807 --> 00:05:20.350 align:middle line:84% to do that over spatial encoding. 00:05:20.350 --> 00:05:21.722 align:middle line:90% So you can work over time. 00:05:21.722 --> 00:05:22.930 align:middle line:90% You can also work over space. 00:05:22.930 --> 00:05:25.240 align:middle line:84% The Bayer filter image, I think this is the exact image 00:05:25.240 --> 00:05:28.450 align:middle line:90% that you guys talked about. 00:05:28.450 --> 00:05:30.610 align:middle line:84% People have also done it with polarization filters. 00:05:30.610 --> 00:05:33.790 align:middle line:84% And then I could also talk about assorted pixels, which 00:05:33.790 --> 00:05:34.930 align:middle line:90% was from [INAUDIBLE] group. 00:05:34.930 --> 00:05:37.690 align:middle line:84% And we mix and match neutral density 00:05:37.690 --> 00:05:39.880 align:middle line:84% and polarization and spectral filtering, so-- 00:05:39.880 --> 00:05:41.140 align:middle line:90% all over a sensor. 00:05:41.140 --> 00:05:46.450 align:middle line:84% The problem with that is it's extremely hard to fabricate. 00:05:46.450 --> 00:05:49.370 align:middle line:84% And once you make it, you're stuck with it. 00:05:49.370 --> 00:05:52.000 align:middle line:84% So you can't very easily take off your lens 00:05:52.000 --> 00:05:55.450 align:middle line:84% and peel off your filters and then put another one on it. 00:05:55.450 --> 00:05:58.660 align:middle line:84% Very sensitive to alignment and stuff. 00:05:58.660 --> 00:06:02.440 align:middle line:84% So other people have tried capturing 00:06:02.440 --> 00:06:04.960 align:middle line:84% multispectral information or other types of information 00:06:04.960 --> 00:06:06.850 align:middle line:90% with just many cameras. 00:06:06.850 --> 00:06:08.690 align:middle line:84% So you guys saw the profusion camera, 00:06:08.690 --> 00:06:10.420 align:middle line:90% we had on the first day. 00:06:10.420 --> 00:06:13.510 align:middle line:84% This is a very simple, just similar design 00:06:13.510 --> 00:06:14.960 align:middle line:90% called periodic or [INAUDIBLE]. 00:06:14.960 --> 00:06:17.470 align:middle line:84% We also learned about where you just put different color 00:06:17.470 --> 00:06:19.870 align:middle line:84% filters over each little camera and you 00:06:19.870 --> 00:06:22.850 align:middle line:84% can get a single snapshot from all the cameras. 00:06:22.850 --> 00:06:25.390 align:middle line:84% The problem with this is there's issues 00:06:25.390 --> 00:06:28.060 align:middle line:84% of registration and alignment with the cameras 00:06:28.060 --> 00:06:29.320 align:middle line:90% and it's expensive. 00:06:29.320 --> 00:06:33.340 align:middle line:90% That camera costs $10,000. 00:06:33.340 --> 00:06:36.790 align:middle line:84% So it's a good method, but our method is a little bit 00:06:36.790 --> 00:06:39.430 align:middle line:90% different, I guess. 00:06:39.430 --> 00:06:41.980 align:middle line:84% The third way, I think the coolest way, of doing this 00:06:41.980 --> 00:06:45.000 align:middle line:84% is with this method called co-division multiplexing. 00:06:45.000 --> 00:06:46.750 align:middle line:84% There's only a couple of examples of this. 00:06:46.750 --> 00:06:49.107 align:middle line:84% But basically, you computationally 00:06:49.107 --> 00:06:51.440 align:middle line:84% capture a three-dimensional data keyed on two dimensions 00:06:51.440 --> 00:06:57.250 align:middle line:84% and then computationally try and guess the third dimension. 00:06:57.250 --> 00:06:59.470 align:middle line:84% So you're trading off in computational power 00:06:59.470 --> 00:07:03.010 align:middle line:84% and also error associated with your estimations. 00:07:03.010 --> 00:07:06.190 align:middle line:84% But some examples of this is this CASSI 00:07:06.190 --> 00:07:08.350 align:middle line:90% which was developed at Duke. 00:07:08.350 --> 00:07:10.450 align:middle line:84% And it's a single snapshot spectral 00:07:10.450 --> 00:07:12.440 align:middle line:90% imager similar to what we do. 00:07:12.440 --> 00:07:15.700 align:middle line:84% What they do is they put a coded mask in an intermediate image 00:07:15.700 --> 00:07:20.350 align:middle line:84% plane and it shifts the data cube in three dimensions. 00:07:20.350 --> 00:07:25.570 align:middle line:84% And they can estimate color, many color channels 00:07:25.570 --> 00:07:27.430 align:middle line:90% from a single image. 00:07:27.430 --> 00:07:30.460 align:middle line:84% And a similar very famous example of that it's called 00:07:30.460 --> 00:07:31.540 align:middle line:90% CTIS. 00:07:31.540 --> 00:07:35.890 align:middle line:84% It stands for, I think, computed tomography infrared 00:07:35.890 --> 00:07:36.700 align:middle line:90% spectrometer. 00:07:36.700 --> 00:07:38.920 align:middle line:90% But it was invented in Arizona. 00:07:38.920 --> 00:07:42.430 align:middle line:84% And what they do is they use a very, very special, 00:07:42.430 --> 00:07:46.720 align:middle line:90% cool holographic plate. 00:07:46.720 --> 00:07:48.370 align:middle line:84% And with that you can take an image, 00:07:48.370 --> 00:07:50.578 align:middle line:84% and the holographic plate makes this very interesting 00:07:50.578 --> 00:07:53.830 align:middle line:84% spectral description, which they know exactly how it's 00:07:53.830 --> 00:07:59.320 align:middle line:84% being dispersed so they can reverse the spectral dispersion 00:07:59.320 --> 00:08:01.570 align:middle line:84% to get a full dimensional image but also 00:08:01.570 --> 00:08:03.940 align:middle line:90% have all this color information. 00:08:03.940 --> 00:08:06.730 align:middle line:84% Problem with it is that the hologram they use 00:08:06.730 --> 00:08:09.110 align:middle line:84% is very sensitive to directionality of light. 00:08:09.110 --> 00:08:12.000 align:middle line:84% So it has a very, very narrow field of view. 00:08:12.000 --> 00:08:13.750 align:middle line:84% So basically they have a really cool movie 00:08:13.750 --> 00:08:15.970 align:middle line:84% of some guy lighting a lighter and you 00:08:15.970 --> 00:08:17.450 align:middle line:90% see all this spectral stuff. 00:08:17.450 --> 00:08:19.117 align:middle line:84% But that says why does the field of view 00:08:19.117 --> 00:08:21.250 align:middle line:90% is it's like one little lighter. 00:08:21.250 --> 00:08:24.700 align:middle line:90% But it's a really cool. 00:08:24.700 --> 00:08:27.933 align:middle line:84% So just the basic idea of our approach-- 00:08:27.933 --> 00:08:29.350 align:middle line:84% I won't get into too many details. 00:08:29.350 --> 00:08:31.160 align:middle line:84% I'll go over this kind of quickly. 00:08:31.160 --> 00:08:32.799 align:middle line:84% But you have a camera, and you're 00:08:32.799 --> 00:08:36.460 align:middle line:84% capturing two spots on an object, on one red one. 00:08:36.460 --> 00:08:38.799 align:middle line:90% And at the center, the rays-- 00:08:38.799 --> 00:08:42.070 align:middle line:84% let's pick the red spot-- are both integrated at the sensor 00:08:42.070 --> 00:08:47.950 align:middle line:84% so you can't tell which ray came from where, 00:08:47.950 --> 00:08:49.510 align:middle line:90% any other part of the lens. 00:08:49.510 --> 00:08:53.890 align:middle line:84% If you misfocus your sensor, you can tell, once again. 00:08:53.890 --> 00:08:57.040 align:middle line:84% So you know this ray is not integrated with that one. 00:08:57.040 --> 00:08:59.920 align:middle line:84% So you know it's coming from this spot in the lines, 00:08:59.920 --> 00:09:00.913 align:middle line:90% likewise for that one. 00:09:00.913 --> 00:09:03.580 align:middle line:84% But when you misfocus something, everything just blurs together. 00:09:03.580 --> 00:09:06.480 align:middle line:84% There's no separation, spatially, of the rays. 00:09:06.480 --> 00:09:09.040 align:middle line:84% It's hard to distinguish them unless you're 00:09:09.040 --> 00:09:12.370 align:middle line:84% doing astronomical imaging where the stars are surrounded 00:09:12.370 --> 00:09:16.440 align:middle line:90% by black and you have a chance. 00:09:16.440 --> 00:09:21.160 align:middle line:84% What we did is we put a pinhole array on top of the sensor. 00:09:21.160 --> 00:09:23.160 align:middle line:84% And that has a couple of interesting properties. 00:09:23.160 --> 00:09:24.618 align:middle line:84% It's very similar to a light field. 00:09:24.618 --> 00:09:25.830 align:middle line:90% I won't really get into it. 00:09:25.830 --> 00:09:27.540 align:middle line:84% But basically what it allows you to do 00:09:27.540 --> 00:09:29.690 align:middle line:84% is it allows you to take your Bayer filter that you 00:09:29.690 --> 00:09:31.440 align:middle line:84% originally had at the sensor [INAUDIBLE],, 00:09:31.440 --> 00:09:33.550 align:middle line:84% and you just move it up into the lens. 00:09:33.550 --> 00:09:36.660 align:middle line:84% So now instead of having a repeated Bayer filter pattern 00:09:36.660 --> 00:09:40.583 align:middle line:84% over your focal point, you just have one filter 00:09:40.583 --> 00:09:42.000 align:middle line:84% and you can stick it in your lens. 00:09:42.000 --> 00:09:43.625 align:middle line:84% And what each of the pinholes are doing 00:09:43.625 --> 00:09:46.740 align:middle line:84% is they're effectively imaging this aperture. 00:09:46.740 --> 00:09:49.480 align:middle line:84% So you can think of each of these little pinhole camera. 00:09:49.480 --> 00:09:51.780 align:middle line:84% And each of them, all you can see is this filter. 00:09:51.780 --> 00:09:54.150 align:middle line:84% So you're creating a style of filter. 00:09:54.150 --> 00:09:56.760 align:middle line:84% But it's also still capturing the information 00:09:56.760 --> 00:09:59.350 align:middle line:90% about the object. 00:09:59.350 --> 00:10:01.350 align:middle line:84% So for example, if you put a red filter 00:10:01.350 --> 00:10:03.630 align:middle line:84% in this part of the ray and a blue filter there, 00:10:03.630 --> 00:10:06.480 align:middle line:84% the red filter will attenuate the blue ray. 00:10:06.480 --> 00:10:08.460 align:middle line:84% So you'll figure out it's red there. 00:10:08.460 --> 00:10:11.070 align:middle line:90% But this is a grayscale sensor. 00:10:11.070 --> 00:10:13.920 align:middle line:84% Only one of the sensor areas will light up. 00:10:13.920 --> 00:10:18.300 align:middle line:84% The same with the [INAUDIBLE] area on the other pinhole. 00:10:18.300 --> 00:10:21.210 align:middle line:90% 00:10:21.210 --> 00:10:23.310 align:middle line:90% So that's the general idea. 00:10:23.310 --> 00:10:25.050 align:middle line:84% But the trade-up with this is now 00:10:25.050 --> 00:10:27.310 align:middle line:84% you have a reduced spatial resolution. 00:10:27.310 --> 00:10:29.310 align:middle line:84% And your spatial resolution of your output image 00:10:29.310 --> 00:10:31.227 align:middle line:84% is going to be given by the number of pinholes 00:10:31.227 --> 00:10:32.760 align:middle line:90% you have in your pinhole there. 00:10:32.760 --> 00:10:35.972 align:middle line:84% And your filter resolution or whatever-- 00:10:35.972 --> 00:10:37.680 align:middle line:84% it's not really color resolution anymore, 00:10:37.680 --> 00:10:39.420 align:middle line:84% because we're going to put different types of filters 00:10:39.420 --> 00:10:41.837 align:middle line:84% in here-- it's going to mean even by the number of filters 00:10:41.837 --> 00:10:43.320 align:middle line:90% you stick in your filter. 00:10:43.320 --> 00:10:45.870 align:middle line:84% And there's an interesting trade-off there, 00:10:45.870 --> 00:10:47.910 align:middle line:84% which I might have time to talk about. 00:10:47.910 --> 00:10:50.910 align:middle line:84% I won't really talk about the specifics. 00:10:50.910 --> 00:10:54.390 align:middle line:84% Basically, just since we have a digital image now 00:10:54.390 --> 00:10:57.150 align:middle line:84% on a film set up, this wouldn't work at all, 00:10:57.150 --> 00:10:59.700 align:middle line:84% because you can't really cut out the little pieces 00:10:59.700 --> 00:11:01.980 align:middle line:84% of different spectral areas and stitch them together. 00:11:01.980 --> 00:11:04.470 align:middle line:84% It'd be really difficult. You could reverse 00:11:04.470 --> 00:11:07.600 align:middle line:84% project it, which people have done with them. 00:11:07.600 --> 00:11:10.300 align:middle line:84% But anyway, we just take different areas. 00:11:10.300 --> 00:11:15.240 align:middle line:84% So let's say-- and this is just a piece of an image, which 00:11:15.240 --> 00:11:17.230 align:middle line:90% I'll show you later. 00:11:17.230 --> 00:11:19.350 align:middle line:84% And each of these is not projecting well. 00:11:19.350 --> 00:11:21.420 align:middle line:84% Each of these white boxes is the area 00:11:21.420 --> 00:11:23.130 align:middle line:84% where the filters are being projected. 00:11:23.130 --> 00:11:26.850 align:middle line:84% So there's a 3 by 3 filter at the center one, I think. 00:11:26.850 --> 00:11:30.400 align:middle line:90% It's a very non-dense filtering. 00:11:30.400 --> 00:11:32.850 align:middle line:84% So let's say this is a green filter from this area. 00:11:32.850 --> 00:11:34.512 align:middle line:84% You can take all these green filters, 00:11:34.512 --> 00:11:36.470 align:middle line:84% stitch them together, and make the green image. 00:11:36.470 --> 00:11:38.460 align:middle line:84% So let's say this is the red filter here. 00:11:38.460 --> 00:11:40.648 align:middle line:84% You can do that in the red image. 00:11:40.648 --> 00:11:41.940 align:middle line:90% And you just do that digitally. 00:11:41.940 --> 00:11:43.230 align:middle line:90% It's just like a lookup table. 00:11:43.230 --> 00:11:47.520 align:middle line:90% It's very straightforward. 00:11:47.520 --> 00:11:49.740 align:middle line:84% And this is what our setup looked like. 00:11:49.740 --> 00:11:51.240 align:middle line:90% We just had a regular lens. 00:11:51.240 --> 00:11:52.300 align:middle line:90% It's a Nikon lens. 00:11:52.300 --> 00:11:53.730 align:middle line:90% It was $100. 00:11:53.730 --> 00:11:55.570 align:middle line:84% And they're really easy to take apart. 00:11:55.570 --> 00:11:57.930 align:middle line:84% So if any of you ever wants for whatever, 00:11:57.930 --> 00:12:00.240 align:middle line:84% your project for this class, to put stuff 00:12:00.240 --> 00:12:02.460 align:middle line:84% in the aperture of a lens, come to me. 00:12:02.460 --> 00:12:04.350 align:middle line:84% I can teach you how to do it in five minutes. 00:12:04.350 --> 00:12:07.770 align:middle line:84% And it'll take you 30 seconds each time to undo the lens 00:12:07.770 --> 00:12:10.090 align:middle line:90% and put it back together. 00:12:10.090 --> 00:12:12.600 align:middle line:84% But we put color filters in there. 00:12:12.600 --> 00:12:14.100 align:middle line:84% You can't really see them that well. 00:12:14.100 --> 00:12:16.860 align:middle line:84% But we just put these little color filters discs. 00:12:16.860 --> 00:12:18.990 align:middle line:84% And then I'll show you some results for something 00:12:18.990 --> 00:12:19.740 align:middle line:90% I did this summer. 00:12:19.740 --> 00:12:22.390 align:middle line:84% I put a spectral filter in the aperture, 00:12:22.390 --> 00:12:26.550 align:middle line:84% which changes our design to a snapshot spectral imager. 00:12:26.550 --> 00:12:28.950 align:middle line:84% And then here we have our [INAUDIBLE].. 00:12:28.950 --> 00:12:31.365 align:middle line:90% This is the CCD. 00:12:31.365 --> 00:12:34.620 align:middle line:84% You just sandwich a pinhole array right up against the CCD. 00:12:34.620 --> 00:12:37.680 align:middle line:84% And the covered glass over the CCD, which is like a millimeter 00:12:37.680 --> 00:12:41.340 align:middle line:84% thick, dictates the distance between the two. 00:12:41.340 --> 00:12:43.787 align:middle line:90% 00:12:43.787 --> 00:12:45.120 align:middle line:90% So here are some example images. 00:12:45.120 --> 00:12:48.868 align:middle line:84% This is just a image, a raw data image. 00:12:48.868 --> 00:12:50.160 align:middle line:90% And here's a zoomed up version. 00:12:50.160 --> 00:12:54.960 align:middle line:84% You can see each little pinhole region close up. 00:12:54.960 --> 00:12:57.210 align:middle line:84% And then you could do some [INAUDIBLE] division, which 00:12:57.210 --> 00:13:00.780 align:middle line:90% is just normalizing the image. 00:13:00.780 --> 00:13:03.750 align:middle line:84% So basically we wanted to use a lenslet array, which 00:13:03.750 --> 00:13:07.160 align:middle line:84% is just almost the same thing as a pinhole array. 00:13:07.160 --> 00:13:08.910 align:middle line:84% I could show you some slides from Stanford 00:13:08.910 --> 00:13:10.327 align:middle line:84% where they have the lenslet array. 00:13:10.327 --> 00:13:11.650 align:middle line:90% That's just really expensive. 00:13:11.650 --> 00:13:14.070 align:middle line:90% And once you get it, it's fixed. 00:13:14.070 --> 00:13:16.170 align:middle line:84% Pinhole arrays you can print off, 00:13:16.170 --> 00:13:18.330 align:middle line:84% you can vary the distances between pinholes 00:13:18.330 --> 00:13:19.690 align:middle line:90% and the sizes of pinholes. 00:13:19.690 --> 00:13:22.320 align:middle line:84% So it's a much easier way to check when 00:13:22.320 --> 00:13:23.850 align:middle line:90% you're doing an experiment. 00:13:23.850 --> 00:13:27.240 align:middle line:84% But they have imperfections, obviously, especially 00:13:27.240 --> 00:13:28.060 align:middle line:90% from printing. 00:13:28.060 --> 00:13:30.450 align:middle line:84% So that's why we did that version of it, sort of get rid 00:13:30.450 --> 00:13:32.468 align:middle line:90% of those imperfections. 00:13:32.468 --> 00:13:34.260 align:middle line:84% And so this was done on a grayscale sensor. 00:13:34.260 --> 00:13:38.550 align:middle line:84% So it's just a gray sensor, no color information. 00:13:38.550 --> 00:13:40.860 align:middle line:84% But we put six filters in the aperture 00:13:40.860 --> 00:13:44.370 align:middle line:84% for this picture, a red, green, and blue filter and then three 00:13:44.370 --> 00:13:45.630 align:middle line:90% polarizers. 00:13:45.630 --> 00:13:47.070 align:middle line:90% So we took an image. 00:13:47.070 --> 00:13:50.010 align:middle line:84% And the images of-- there's a big TV back here. 00:13:50.010 --> 00:13:52.920 align:middle line:84% And there's a color chart here and a book 00:13:52.920 --> 00:13:55.360 align:middle line:84% and then a polarizer sheet over here. 00:13:55.360 --> 00:13:58.110 align:middle line:84% But you can see we've got a color image, which 00:13:58.110 --> 00:14:00.115 align:middle line:84% approximates the color of the scene pretty well. 00:14:00.115 --> 00:14:00.615 align:middle line:90% Yes. 00:14:00.615 --> 00:14:02.640 align:middle line:84% AUDIENCE: When you say "printed", 00:14:02.640 --> 00:14:04.986 align:middle line:90% why you say "printed"? 00:14:04.986 --> 00:14:08.100 align:middle line:84% ROARKE HORSTMEYER: So there's this company called Pageworks. 00:14:08.100 --> 00:14:09.750 align:middle line:90% It's actually in Cambridge. 00:14:09.750 --> 00:14:12.240 align:middle line:90% And, what's the process called? 00:14:12.240 --> 00:14:13.320 align:middle line:90% I'm blanking. 00:14:13.320 --> 00:14:17.585 align:middle line:84% AUDIENCE: Because it's just really just [INAUDIBLE] 00:14:17.585 --> 00:14:19.680 align:middle line:84% ROARKE HORSTMEYER: So it's on our transparency. 00:14:19.680 --> 00:14:21.170 align:middle line:84% So it's just like printing a transparency. 00:14:21.170 --> 00:14:21.460 align:middle line:90% AUDIENCE: OK. 00:14:21.460 --> 00:14:21.660 align:middle line:90% ROARKE HORSTMEYER: Yeah. 00:14:21.660 --> 00:14:22.650 align:middle line:90% PROFESSOR: OK. 00:14:22.650 --> 00:14:25.440 align:middle line:90% It's just easy to make. 00:14:25.440 --> 00:14:27.120 align:middle line:84% And the contrast isn't great, but it's 00:14:27.120 --> 00:14:30.242 align:middle line:90% because there is, [INAUDIBLE] 00:14:30.242 --> 00:14:31.980 align:middle line:90% ROARKE HORSTMEYER: Yeah, yeah. 00:14:31.980 --> 00:14:34.333 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:14:34.333 --> 00:14:36.000 align:middle line:84% ROARKE HORSTMEYER: No, it wouldn't have. 00:14:36.000 --> 00:14:39.240 align:middle line:84% So the pinhole sizes are about 50 microns. 00:14:39.240 --> 00:14:40.455 align:middle line:90% So I don't think the laser-- 00:14:40.455 --> 00:14:44.666 align:middle line:90% AUDIENCE: [INAUDIBLE] shrink. 00:14:44.666 --> 00:14:48.850 align:middle line:84% AUDIENCE: So that's what we were trying to do last year. 00:14:48.850 --> 00:14:50.290 align:middle line:90% AUDIENCE: And then, what? 00:14:50.290 --> 00:14:51.910 align:middle line:90% AUDIENCE: Well, it does work. 00:14:51.910 --> 00:14:52.507 align:middle line:90% AUDIENCE: OK. 00:14:52.507 --> 00:14:55.090 align:middle line:84% AUDIENCE: For what we wanted to do, we needed still much more. 00:14:55.090 --> 00:14:56.270 align:middle line:90% AUDIENCE: So it's sort of-- 00:14:56.270 --> 00:14:56.770 align:middle line:90% OK. 00:14:56.770 --> 00:14:58.062 align:middle line:90% AUDIENCE: But, yeah, it worked. 00:14:58.062 --> 00:14:59.840 align:middle line:84% You print on a [INAUDIBLE] and then-- 00:14:59.840 --> 00:15:01.970 align:middle line:84% AUDIENCE: And it sort of-- it shrinks and it-- 00:15:01.970 --> 00:15:03.315 align:middle line:90% AUDIENCE: It's not uniform but-- 00:15:03.315 --> 00:15:04.620 align:middle line:90% AUDIENCE: OK. 00:15:04.620 --> 00:15:05.620 align:middle line:90% ROARKE HORSTMEYER: Yeah. 00:15:05.620 --> 00:15:07.453 align:middle line:84% So yeah, you just print out on transparency. 00:15:07.453 --> 00:15:10.570 align:middle line:84% But you have to use, you can't just do it on a normal printer. 00:15:10.570 --> 00:15:12.700 align:middle line:84% You need a high resolution printer. 00:15:12.700 --> 00:15:13.450 align:middle line:90% Well, we did. 00:15:13.450 --> 00:15:16.660 align:middle line:84% Because we needed 25 micron resolution. 00:15:16.660 --> 00:15:19.750 align:middle line:90% 720, I think, dpi. 00:15:19.750 --> 00:15:22.540 align:middle line:84% But anyway, so in one image, we get a color picture. 00:15:22.540 --> 00:15:25.330 align:middle line:84% But we also get a degree of color polarization picture. 00:15:25.330 --> 00:15:26.950 align:middle line:84% So you can see the TV's polarized, 00:15:26.950 --> 00:15:28.877 align:middle line:84% because TVs have polarizers in them. 00:15:28.877 --> 00:15:30.460 align:middle line:84% There, you can see down there, there's 00:15:30.460 --> 00:15:32.590 align:middle line:84% a polarized down there over a resolution chart. 00:15:32.590 --> 00:15:35.457 align:middle line:90% 00:15:35.457 --> 00:15:36.790 align:middle line:90% So that was a simple little tip. 00:15:36.790 --> 00:15:39.760 align:middle line:90% Then we did one of 16 filters. 00:15:39.760 --> 00:15:42.250 align:middle line:84% So basically I bought every filter I could 00:15:42.250 --> 00:15:45.710 align:middle line:84% from Edmonds that was cheap and put them in a filter array. 00:15:45.710 --> 00:15:48.610 align:middle line:84% So we have red, green, blue, yellow, magenta, cyan. 00:15:48.610 --> 00:15:50.330 align:middle line:90% I got an IR filter. 00:15:50.330 --> 00:15:50.830 align:middle line:90% I got a 00:15:50.830 --> 00:15:55.660 align:middle line:84% bunch of different polarizers going to different angles 00:15:55.660 --> 00:15:58.930 align:middle line:84% to get a precise degree of polarization measurement. 00:15:58.930 --> 00:16:01.570 align:middle line:84% And then I put three neutral density filters. 00:16:01.570 --> 00:16:05.620 align:middle line:84% And so these are the 16 images you get from one single image. 00:16:05.620 --> 00:16:08.690 align:middle line:84% You just separate them with a lookup table. 00:16:08.690 --> 00:16:10.717 align:middle line:84% So you can see up here is the color information. 00:16:10.717 --> 00:16:12.550 align:middle line:84% It's kind of hard to see from the projector. 00:16:12.550 --> 00:16:16.240 align:middle line:84% But the spots are the color are changing, 00:16:16.240 --> 00:16:19.690 align:middle line:84% the checkerboard pattern's changing for the color. 00:16:19.690 --> 00:16:20.950 align:middle line:90% The near-IR is really dark. 00:16:20.950 --> 00:16:23.920 align:middle line:84% And I've got a lamp here, which is saturated on some. 00:16:23.920 --> 00:16:27.340 align:middle line:84% But you can see the resolution chart with the near-IR. 00:16:27.340 --> 00:16:30.130 align:middle line:84% The polarization filter, you can see 00:16:30.130 --> 00:16:35.830 align:middle line:84% the TV screen screen's changing and how bright it is and stuff. 00:16:35.830 --> 00:16:38.232 align:middle line:84% And then neutral density is just darker. 00:16:38.232 --> 00:16:40.908 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:16:40.908 --> 00:16:42.200 align:middle line:90% ROARKE HORSTMEYER: What's that? 00:16:42.200 --> 00:16:45.200 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:16:45.200 --> 00:16:47.210 align:middle line:90% ROARKE HORSTMEYER: Yeah, yeah. 00:16:47.210 --> 00:16:50.353 align:middle line:84% AUDIENCE: But then you stack them from each other. 00:16:50.353 --> 00:16:51.770 align:middle line:84% ROARKE HORSTMEYER: No, they're not 00:16:51.770 --> 00:16:53.060 align:middle line:90% stacked on top of each other. 00:16:53.060 --> 00:16:54.865 align:middle line:90% AUDIENCE: Oh. 00:16:54.865 --> 00:16:56.960 align:middle line:84% ROARKE HORSTMEYER: So here are the filters. 00:16:56.960 --> 00:16:58.730 align:middle line:90% It's hard to see on a projector. 00:16:58.730 --> 00:17:02.600 align:middle line:84% But here you can see it's just this little 3 by 3 00:17:02.600 --> 00:17:04.010 align:middle line:90% filter array. 00:17:04.010 --> 00:17:06.270 align:middle line:90% Here you can see. 00:17:06.270 --> 00:17:09.740 align:middle line:84% So basically it's just literally a flat 2D array of filters. 00:17:09.740 --> 00:17:12.050 align:middle line:84% And I put it right on the aperture stop. 00:17:12.050 --> 00:17:14.900 align:middle line:84% You have about this much room on the aperture stop, a little bit 00:17:14.900 --> 00:17:16.460 align:middle line:90% bigger than a quarter or so. 00:17:16.460 --> 00:17:19.160 align:middle line:84% So you can put as many filters in that space as you can. 00:17:19.160 --> 00:17:21.140 align:middle line:84% AUDIENCE: So this also decreases resolution. 00:17:21.140 --> 00:17:22.640 align:middle line:84% ROARKE HORSTMEYER: Yeah, definitely. 00:17:22.640 --> 00:17:23.473 align:middle line:90% [INTERPOSING VOICES] 00:17:23.473 --> 00:17:24.310 align:middle line:90% Definitely. 00:17:24.310 --> 00:17:26.810 align:middle line:84% So your spatial resolution is going 00:17:26.810 --> 00:17:29.840 align:middle line:84% to be decreased at least proportional. 00:17:29.840 --> 00:17:34.490 align:middle line:84% If you did it perfectly and you put one pinhole image 00:17:34.490 --> 00:17:36.950 align:middle line:84% of each filter to 1 pixel it would be decreased 00:17:36.950 --> 00:17:38.880 align:middle line:90% by the number of filters. 00:17:38.880 --> 00:17:39.830 align:middle line:90% So I have 16 filters. 00:17:39.830 --> 00:17:41.330 align:middle line:84% My spatial resolution is going to be 00:17:41.330 --> 00:17:44.930 align:middle line:84% at least 1/16 the original spatial resolution. 00:17:44.930 --> 00:17:47.430 align:middle line:90% That's a critical point. 00:17:47.430 --> 00:17:49.655 align:middle line:84% So these are very low spatial resolution images. 00:17:49.655 --> 00:17:52.170 align:middle line:84% AUDIENCE: The effect is very similar to what they are-- 00:17:52.170 --> 00:17:54.020 align:middle line:90% ROARKE HORSTMEYER: Exactly. 00:17:54.020 --> 00:17:55.940 align:middle line:84% So it's the same thing as a Bayer filter. 00:17:55.940 --> 00:17:58.190 align:middle line:84% You have a red, green, and blue filter. 00:17:58.190 --> 00:17:59.120 align:middle line:90% It's a third. 00:17:59.120 --> 00:18:01.100 align:middle line:84% AUDIENCE: But you get this on the lens, right? 00:18:01.100 --> 00:18:03.530 align:middle line:90% So it's not on the sensor. 00:18:03.530 --> 00:18:07.100 align:middle line:84% So how do you know which actually pixels 00:18:07.100 --> 00:18:09.620 align:middle line:90% are affected by those guys? 00:18:09.620 --> 00:18:12.185 align:middle line:84% ROARKE HORSTMEYER: Yeah, so each image-- 00:18:12.185 --> 00:18:15.260 align:middle line:84% AUDIENCE: Because if you go back to the slide when you show, OK, 00:18:15.260 --> 00:18:16.640 align:middle line:90% this part of the [INAUDIBLE] 00:18:16.640 --> 00:18:17.640 align:middle line:90% ROARKE HORSTMEYER: Yeah. 00:18:17.640 --> 00:18:22.910 align:middle line:84% AUDIENCE: So you can actually know exactly each pixel 00:18:22.910 --> 00:18:25.040 align:middle line:90% was effected by-- 00:18:25.040 --> 00:18:28.270 align:middle line:84% ROARKE HORSTMEYER: Yeah, so I'm sticking a pinhole right 00:18:28.270 --> 00:18:29.860 align:middle line:90% over the sensor. 00:18:29.860 --> 00:18:32.710 align:middle line:84% So the pinhole array is going to create multiple images. 00:18:32.710 --> 00:18:35.110 align:middle line:84% Each one of these dots is a pinhole array image 00:18:35.110 --> 00:18:35.680 align:middle line:90% essentially. 00:18:35.680 --> 00:18:37.090 align:middle line:90% And we zoom into that. 00:18:37.090 --> 00:18:40.720 align:middle line:84% I'm highlighting one pinhole array this white box. 00:18:40.720 --> 00:18:43.060 align:middle line:84% And so I know just from a priori knowledge, 00:18:43.060 --> 00:18:46.570 align:middle line:84% when I put the filter array in there that this center 00:18:46.570 --> 00:18:48.220 align:middle line:90% filter is a red filter. 00:18:48.220 --> 00:18:50.740 align:middle line:84% I know the one on the upper left is a green filter. 00:18:50.740 --> 00:18:52.810 align:middle line:84% The one in the upper right's a blue, for example. 00:18:52.810 --> 00:18:54.890 align:middle line:84% Because I put the filter array in there. 00:18:54.890 --> 00:18:58.690 align:middle line:84% So now under each pinhole, I have a, 00:18:58.690 --> 00:19:00.940 align:middle line:84% ideally I would have nine pixels. 00:19:00.940 --> 00:19:03.580 align:middle line:84% And I would know which each pixel corresponds 00:19:03.580 --> 00:19:06.280 align:middle line:84% to just from the orientation of how I put the filter 00:19:06.280 --> 00:19:07.374 align:middle line:90% array in there. 00:19:07.374 --> 00:19:09.670 align:middle line:84% AUDIENCE: And the [INAUDIBLE] has 00:19:09.670 --> 00:19:11.412 align:middle line:84% to be fixed for this to work, right? 00:19:11.412 --> 00:19:12.370 align:middle line:90% ROARKE HORSTMEYER: Yes. 00:19:12.370 --> 00:19:14.825 align:middle line:90% So I glanced over that. 00:19:14.825 --> 00:19:15.700 align:middle line:90% I skipped that slide. 00:19:15.700 --> 00:19:20.230 align:middle line:84% But yes, you basically have to be imaged at infinity. 00:19:20.230 --> 00:19:23.260 align:middle line:90% You can't change the focus. 00:19:23.260 --> 00:19:26.980 align:middle line:84% Once you change the focus, you're 00:19:26.980 --> 00:19:30.130 align:middle line:84% basically crossing those rays in front of the sensor plane, 00:19:30.130 --> 00:19:33.283 align:middle line:84% and that's destroying your detection diversity. 00:19:33.283 --> 00:19:35.116 align:middle line:84% AUDIENCE: So to scene has to be planned out. 00:19:35.116 --> 00:19:35.950 align:middle line:90% ROARKE HORSTMEYER: Yeah. 00:19:35.950 --> 00:19:37.310 align:middle line:84% AUDIENCE: And it has to be [INAUDIBLE] 00:19:37.310 --> 00:19:38.390 align:middle line:84% ROARKE HORSTMEYER: The scene has to be planned. 00:19:38.390 --> 00:19:39.200 align:middle line:90% And lambertian. 00:19:39.200 --> 00:19:41.090 align:middle line:84% Well, not technically lambertian. 00:19:41.090 --> 00:19:45.597 align:middle line:84% It has to emit polarization or spectral information. 00:19:45.597 --> 00:19:47.180 align:middle line:84% AUDIENCE: But they should be the same. 00:19:47.180 --> 00:19:50.100 align:middle line:84% ROARKE HORSTMEYER: And all the same rays. 00:19:50.100 --> 00:19:53.120 align:middle line:84% AUDIENCE: Did you try removing the effect of reducing 00:19:53.120 --> 00:19:56.160 align:middle line:84% the spatial resolution by trying to demodulate? 00:19:56.160 --> 00:19:58.970 align:middle line:84% Because see, the pinhole array is basically a [INAUDIBLE] 00:19:58.970 --> 00:20:02.980 align:middle line:84% which has a modulation function on it. 00:20:02.980 --> 00:20:06.020 align:middle line:84% So if we demodulate it on computers 00:20:06.020 --> 00:20:08.040 align:middle line:84% once the image has been captured, 00:20:08.040 --> 00:20:10.370 align:middle line:84% you can actually regain the spatial resolution 00:20:10.370 --> 00:20:12.344 align:middle line:90% reduced by modulation. 00:20:12.344 --> 00:20:14.845 align:middle line:90% 00:20:14.845 --> 00:20:16.220 align:middle line:84% ROARKE HORSTMEYER: I don't really 00:20:16.220 --> 00:20:18.950 align:middle line:90% know what you mean modulation. 00:20:18.950 --> 00:20:21.320 align:middle line:84% Basically, since you have filters in your aperture, 00:20:21.320 --> 00:20:24.560 align:middle line:90% those are attenuating light. 00:20:24.560 --> 00:20:26.690 align:middle line:84% AUDIENCE: Operating a function over the image. 00:20:26.690 --> 00:20:28.357 align:middle line:84% ROARKE HORSTMEYER: Yeah, but you're also 00:20:28.357 --> 00:20:29.900 align:middle line:90% mixing in all this diversity. 00:20:29.900 --> 00:20:32.660 align:middle line:84% So now you have all this spectral and polarization 00:20:32.660 --> 00:20:35.240 align:middle line:84% diversity mixed in from the different filters, which 00:20:35.240 --> 00:20:38.540 align:middle line:84% are going to change depending on your spatial location. 00:20:38.540 --> 00:20:40.400 align:middle line:84% So if I have a polarized object up here, 00:20:40.400 --> 00:20:45.000 align:middle line:84% it'll be modulated by 1 polarizer and not by another. 00:20:45.000 --> 00:20:47.690 align:middle line:84% PROFESSOR: It's just like there [INAUDIBLE].. 00:20:47.690 --> 00:20:53.340 align:middle line:84% You can take off some sophisticated [INAUDIBLE] 00:20:53.340 --> 00:20:53.900 align:middle line:90% information. 00:20:53.900 --> 00:20:57.250 align:middle line:84% But it wouldn't just be hard [INAUDIBLE] 00:20:57.250 --> 00:20:58.250 align:middle line:90% ROARKE HORSTMEYER: Yeah. 00:20:58.250 --> 00:21:01.820 align:middle line:90% 00:21:01.820 --> 00:21:02.450 align:middle line:90% OK. 00:21:02.450 --> 00:21:03.860 align:middle line:90% I'll just try to finish up. 00:21:03.860 --> 00:21:07.430 align:middle line:84% So just an example of using those 16 images. 00:21:07.430 --> 00:21:09.602 align:middle line:84% People ask, well, why do you want 16 images of all 00:21:09.602 --> 00:21:10.560 align:middle line:90% these different things? 00:21:10.560 --> 00:21:13.000 align:middle line:84% So I try to come up with a clever example. 00:21:13.000 --> 00:21:15.380 align:middle line:84% But basically you can create a color image 00:21:15.380 --> 00:21:18.740 align:middle line:84% and you have this huge saturated area from the sun 00:21:18.740 --> 00:21:20.030 align:middle line:90% or from a lamp or whatever. 00:21:20.030 --> 00:21:25.760 align:middle line:84% So you can just make a quick HDR image using the three HDR 00:21:25.760 --> 00:21:27.750 align:middle line:84% filters on top of the color information. 00:21:27.750 --> 00:21:31.190 align:middle line:84% So now you can see there's like a resolution chart back there. 00:21:31.190 --> 00:21:33.080 align:middle line:84% And then what you can do is take a degree 00:21:33.080 --> 00:21:35.660 align:middle line:84% of polarization measurement using the five polarization 00:21:35.660 --> 00:21:36.260 align:middle line:90% filters. 00:21:36.260 --> 00:21:39.170 align:middle line:84% And that'll find things that are reflecting polarized light 00:21:39.170 --> 00:21:40.490 align:middle line:90% or emitting polarized light. 00:21:40.490 --> 00:21:43.947 align:middle line:84% So for example, the man-made object example 00:21:43.947 --> 00:21:44.780 align:middle line:90% I showed you before. 00:21:44.780 --> 00:21:47.960 align:middle line:84% You can pick out which objects are man-made, which are these 00:21:47.960 --> 00:21:48.870 align:middle line:90% or whatever. 00:21:48.870 --> 00:21:51.770 align:middle line:84% And you can also pick out things emitting IR information 00:21:51.770 --> 00:21:52.850 align:middle line:90% in the IR filter. 00:21:52.850 --> 00:21:55.100 align:middle line:84% So people or well, I guess plants 00:21:55.100 --> 00:21:56.940 align:middle line:90% emit near-IR more than people. 00:21:56.940 --> 00:21:59.600 align:middle line:84% But if it was extended into the IR range, 00:21:59.600 --> 00:22:01.550 align:middle line:90% you could identify people. 00:22:01.550 --> 00:22:06.650 align:middle line:84% But you would need a better sense than a regular CCD. 00:22:06.650 --> 00:22:08.540 align:middle line:84% So you combine all those and I did it 00:22:08.540 --> 00:22:10.850 align:middle line:84% with 12 different filters out of the 16. 00:22:10.850 --> 00:22:15.200 align:middle line:84% And I created this sort of phony region of interest extended, 00:22:15.200 --> 00:22:18.620 align:middle line:90% dynamic range color image. 00:22:18.620 --> 00:22:21.680 align:middle line:84% And you can see there's some errors, obviously. 00:22:21.680 --> 00:22:24.110 align:middle line:84% And that's because what I was trying to find polarization, 00:22:24.110 --> 00:22:26.120 align:middle line:84% for example, if it's a low intensity area, 00:22:26.120 --> 00:22:29.030 align:middle line:84% it's not going to emit any real information about polarization. 00:22:29.030 --> 00:22:30.830 align:middle line:84% It's just going to be a dark area. 00:22:30.830 --> 00:22:33.350 align:middle line:90% Things like that causing error. 00:22:33.350 --> 00:22:38.180 align:middle line:84% And also from angular or the different planes perspective, 00:22:38.180 --> 00:22:41.250 align:middle line:84% you can see that here that there's a bar sticking out. 00:22:41.250 --> 00:22:44.540 align:middle line:84% And that's because as things get closer, 00:22:44.540 --> 00:22:47.510 align:middle line:84% error associated with essentially 00:22:47.510 --> 00:22:50.330 align:middle line:84% having different perspectives on an object becomes apparent. 00:22:50.330 --> 00:22:52.940 align:middle line:84% So it only works for plane or objects towards infinity, 00:22:52.940 --> 00:22:53.780 align:middle line:90% or it works best. 00:22:53.780 --> 00:22:57.135 align:middle line:90% 00:22:57.135 --> 00:22:58.760 align:middle line:84% So the next example, this is I just did 00:22:58.760 --> 00:23:01.340 align:middle line:84% this over the summer for a quick paper. 00:23:01.340 --> 00:23:03.020 align:middle line:84% Instead of putting a filter array, 00:23:03.020 --> 00:23:07.368 align:middle line:84% we put a spectral filter in the aperture down here. 00:23:07.368 --> 00:23:09.410 align:middle line:84% And what this does is essentially like a grating, 00:23:09.410 --> 00:23:10.880 align:middle line:84% just splits up the incoming light 00:23:10.880 --> 00:23:13.190 align:middle line:90% into all different wavelengths. 00:23:13.190 --> 00:23:15.510 align:middle line:84% It went from 400 to 700 nanometers, 00:23:15.510 --> 00:23:17.270 align:middle line:90% so across the color spectrum. 00:23:17.270 --> 00:23:21.585 align:middle line:84% And it had a resolution of about 10 nanometers. 00:23:21.585 --> 00:23:22.460 align:middle line:90% That was pretty good. 00:23:22.460 --> 00:23:23.210 align:middle line:90% And they're cheap. 00:23:23.210 --> 00:23:25.370 align:middle line:90% They're like 200 bucks. 00:23:25.370 --> 00:23:28.040 align:middle line:84% And they're small enough to fit in an aperture. 00:23:28.040 --> 00:23:30.820 align:middle line:84% So they're really useful to do experiments with cameras. 00:23:30.820 --> 00:23:33.410 align:middle line:84% And so I just took a picture of some crayons. 00:23:33.410 --> 00:23:34.650 align:middle line:90% And picking out different-- 00:23:34.650 --> 00:23:37.070 align:middle line:84% So this is a reduced-resolution image of some crayons. 00:23:37.070 --> 00:23:38.780 align:middle line:84% By picking out certain pixels, you 00:23:38.780 --> 00:23:44.300 align:middle line:84% can get a roughly 25-pixel resolution of the 400 to 700 00:23:44.300 --> 00:23:45.510 align:middle line:90% nanometer range. 00:23:45.510 --> 00:23:46.910 align:middle line:90% So you can see this is magenta. 00:23:46.910 --> 00:23:48.815 align:middle line:84% That's this red and blue parts, but not 00:23:48.815 --> 00:23:51.920 align:middle line:90% the green, teal, orange, blue. 00:23:51.920 --> 00:23:55.060 align:middle line:90% 00:23:55.060 --> 00:23:58.510 align:middle line:84% So yeah, I have one more example of that. 00:23:58.510 --> 00:24:01.617 align:middle line:84% So this is same crayons, different pictures. 00:24:01.617 --> 00:24:03.450 align:middle line:84% So I noticed when I was doing the experiment 00:24:03.450 --> 00:24:07.260 align:middle line:84% that if I had the lights on or if I had another light on, 00:24:07.260 --> 00:24:09.112 align:middle line:84% I would just get really different results. 00:24:09.112 --> 00:24:10.320 align:middle line:90% And I was scratching my head. 00:24:10.320 --> 00:24:11.195 align:middle line:90% Why does that happen? 00:24:11.195 --> 00:24:13.920 align:middle line:84% Well, it's obviously because fluorescent lights emit so much 00:24:13.920 --> 00:24:16.440 align:middle line:90% different spectral information. 00:24:16.440 --> 00:24:19.230 align:middle line:84% Or I just can't really tell, but these lights 00:24:19.230 --> 00:24:21.630 align:middle line:84% versus these fluorescent lights have a very different 00:24:21.630 --> 00:24:24.220 align:middle line:90% spectral distribution. 00:24:24.220 --> 00:24:25.620 align:middle line:90% So I didn't really label them. 00:24:25.620 --> 00:24:28.658 align:middle line:84% But these are definitely just fluorescent lights. 00:24:28.658 --> 00:24:30.450 align:middle line:84% Or maybe one's fluorescent lights and one's 00:24:30.450 --> 00:24:32.160 align:middle line:84% another desktop fluorescent light. 00:24:32.160 --> 00:24:34.200 align:middle line:84% So you can see the sharp peak throug this. 00:24:34.200 --> 00:24:36.140 align:middle line:84% Most pixels just have a sharp peak. 00:24:36.140 --> 00:24:37.882 align:middle line:84% Our eyes just integrate over all of that. 00:24:37.882 --> 00:24:39.090 align:middle line:90% So we don't really notice it. 00:24:39.090 --> 00:24:40.510 align:middle line:90% Just looks like light. 00:24:40.510 --> 00:24:42.270 align:middle line:84% But if I just put on a regular desk lamp, 00:24:42.270 --> 00:24:46.030 align:middle line:84% like an old fashioned bulb, you get this distribution. 00:24:46.030 --> 00:24:48.330 align:middle line:84% So these are all from the same exact pixel. 00:24:48.330 --> 00:24:50.460 align:middle line:84% But I was just changing the lights in the room. 00:24:50.460 --> 00:24:53.152 align:middle line:90% So it was pretty interesting. 00:24:53.152 --> 00:24:57.922 align:middle line:84% PROFESSOR: [INAUDIBLE] the light coming in as a modification 00:24:57.922 --> 00:24:58.630 align:middle line:90% of the incident-- 00:24:58.630 --> 00:25:00.130 align:middle line:84% ROARKE HORSTMEYER: Incident lighting 00:25:00.130 --> 00:25:02.030 align:middle line:90% and the color of the-- exactly. 00:25:02.030 --> 00:25:03.470 align:middle line:90% Exactly. 00:25:03.470 --> 00:25:08.670 align:middle line:90% So, yeah, that's it. 00:25:08.670 --> 00:25:11.208 align:middle line:90% Any questions? 00:25:11.208 --> 00:25:13.367 align:middle line:84% AUDIENCE: How many [INAUDIBLE] you have? 00:25:13.367 --> 00:25:14.575 align:middle line:90% ROARKE HORSTMEYER: From this? 00:25:14.575 --> 00:25:18.070 align:middle line:90% 00:25:18.070 --> 00:25:19.180 align:middle line:90% Roughly 25. 00:25:19.180 --> 00:25:21.490 align:middle line:84% It just depends on your pinhole array pitch. 00:25:21.490 --> 00:25:24.350 align:middle line:84% Another thing-- so I really want to do this and make a better 00:25:24.350 --> 00:25:24.850 align:middle line:90% filter. 00:25:24.850 --> 00:25:28.180 align:middle line:84% Because the filter I use is just a one-dimensional filter. 00:25:28.180 --> 00:25:31.210 align:middle line:84% It's generally used for laser experiments. 00:25:31.210 --> 00:25:34.270 align:middle line:84% So you hit the laser on it and you can tilt in different ways 00:25:34.270 --> 00:25:37.330 align:middle line:84% and select a specific wavelength if the laser is more 00:25:37.330 --> 00:25:39.590 align:middle line:90% broadband than 10 nanometers. 00:25:39.590 --> 00:25:43.870 align:middle line:90% So it was only one-dimensional. 00:25:43.870 --> 00:25:46.300 align:middle line:84% So it would be cool, much better and more efficient 00:25:46.300 --> 00:25:48.220 align:middle line:84% if I had a two-dimensional filter so 00:25:48.220 --> 00:25:52.660 align:middle line:90% that I could use all the area. 00:25:52.660 --> 00:25:54.430 align:middle line:90% But yeah, roughly 25. 00:25:54.430 --> 00:25:55.600 align:middle line:90% 25. 00:25:55.600 --> 00:25:56.555 align:middle line:90% Any other questions? 00:25:56.555 --> 00:25:59.050 align:middle line:84% AUDIENCE: So those graphs were made out of 25 [INAUDIBLE].. 00:25:59.050 --> 00:26:00.558 align:middle line:84% ROARKE HORSTMEYER: Yeah, yeah, yeah. 00:26:00.558 --> 00:26:04.700 align:middle line:84% AUDIENCE: [INAUDIBLE] in your experiment, 00:26:04.700 --> 00:26:07.225 align:middle line:84% so each pinhole was making an image of the-- 00:26:07.225 --> 00:26:08.850 align:middle line:84% ROARKE HORSTMEYER: Of the filter, yeah. 00:26:08.850 --> 00:26:09.933 align:middle line:90% AUDIENCE: --of the filter? 00:26:09.933 --> 00:26:12.460 align:middle line:84% How much in your experiment, how many of the pixels 00:26:12.460 --> 00:26:14.170 align:middle line:90% were just wasted on-- 00:26:14.170 --> 00:26:15.960 align:middle line:84% ROARKE HORSTMEYER: On the border? 00:26:15.960 --> 00:26:16.510 align:middle line:90% Yeah. 00:26:16.510 --> 00:26:17.110 align:middle line:90% A lot. 00:26:17.110 --> 00:26:18.650 align:middle line:90% A lot. 00:26:18.650 --> 00:26:21.880 align:middle line:84% So I started with a, I think it was 10-megapixel sensor. 00:26:21.880 --> 00:26:23.980 align:middle line:84% And my spatial resolution was around 300 00:26:23.980 --> 00:26:25.870 align:middle line:90% by 300 for each of the images. 00:26:25.870 --> 00:26:30.110 align:middle line:84% So that's for something like 16 filters or nine filters. 00:26:30.110 --> 00:26:32.800 align:middle line:84% So a ton is getting thrown away somewhere. 00:26:32.800 --> 00:26:36.560 align:middle line:84% And each filter is not being imaged to one single pixel. 00:26:36.560 --> 00:26:40.150 align:middle line:84% It's being imaged to a 3 by 3 or 4 by 4 array of pixels. 00:26:40.150 --> 00:26:44.120 align:middle line:84% So I mean it's incredibly wasteful but it's just an idea. 00:26:44.120 --> 00:26:46.270 align:middle line:84% So you can really fine tune it, probably 00:26:46.270 --> 00:26:47.940 align:middle line:90% make it more efficient. 00:26:47.940 --> 00:26:50.107 align:middle line:84% PROFESSOR: I think the big advantage of this, 00:26:50.107 --> 00:26:51.940 align:middle line:84% you can do all of those things while putting 00:26:51.940 --> 00:26:55.690 align:middle line:84% a filter in the lens and not have to put it on the sensor. 00:26:55.690 --> 00:26:58.828 align:middle line:84% So someone was asking, can you change those kinds of things. 00:26:58.828 --> 00:27:00.370 align:middle line:84% Yes, you can just take it out and put 00:27:00.370 --> 00:27:05.020 align:middle line:84% a new one, a different spectral response if you need it. 00:27:05.020 --> 00:27:06.390 align:middle line:90% That's what's interesting. 00:27:06.390 --> 00:27:08.680 align:middle line:90% ROARKE HORSTMEYER: Yeah. 00:27:08.680 --> 00:27:12.270 align:middle line:84% PROFESSOR: Can you say something about the diffraction issues? 00:27:12.270 --> 00:27:13.030 align:middle line:90% Do you have any? 00:27:13.030 --> 00:27:15.100 align:middle line:84% ROARKE HORSTMEYER: Yeah, a little bit. 00:27:15.100 --> 00:27:17.880 align:middle line:84% So some pinholes are really pretty inefficient, pretty 00:27:17.880 --> 00:27:21.720 align:middle line:84% bad at resolving light, which you guys might learn about. 00:27:21.720 --> 00:27:26.260 align:middle line:84% So lenses, everything has an associate. 00:27:26.260 --> 00:27:27.760 align:middle line:84% It's called a point spread function. 00:27:27.760 --> 00:27:29.760 align:middle line:84% I don't if you've heard that term, but basically 00:27:29.760 --> 00:27:32.490 align:middle line:90% a blur spot size. 00:27:32.490 --> 00:27:35.730 align:middle line:84% Pinhole blur spot sizes are more of a geometric projection 00:27:35.730 --> 00:27:37.410 align:middle line:90% of the size of the pinhole. 00:27:37.410 --> 00:27:40.410 align:middle line:84% So you want to let a lot of light through with a pinhole. 00:27:40.410 --> 00:27:42.370 align:middle line:84% But depending on the size of the pinhole, 00:27:42.370 --> 00:27:44.370 align:middle line:84% you can just imagine light just passing straight 00:27:44.370 --> 00:27:48.400 align:middle line:84% through the pinhole and not changing its direction. 00:27:48.400 --> 00:27:51.870 align:middle line:84% So that light will have the blur spot size roughly the size 00:27:51.870 --> 00:27:52.920 align:middle line:90% of the actual pinhole. 00:27:52.920 --> 00:27:54.610 align:middle line:84% So that blur spot size, in my case, 00:27:54.610 --> 00:27:56.130 align:middle line:90% I was using 50 micron pinholes. 00:27:56.130 --> 00:27:59.670 align:middle line:84% It's going to be 50 microns, which is relatively large. 00:27:59.670 --> 00:28:01.410 align:middle line:84% Because a pixel is about 10 microns. 00:28:01.410 --> 00:28:05.725 align:middle line:84% So that's why I was imaging to about five pixels. 00:28:05.725 --> 00:28:08.100 align:middle line:84% If I use the lenslet array, it would have been much, much 00:28:08.100 --> 00:28:10.470 align:middle line:84% better diffraction effects and whatnot would 00:28:10.470 --> 00:28:12.785 align:middle line:90% have been much less. 00:28:12.785 --> 00:28:14.380 align:middle line:84% PROFESSOR: I think we're going to be 00:28:14.380 --> 00:28:17.600 align:middle line:84% talking about using the next array in other projects. 00:28:17.600 --> 00:28:19.600 align:middle line:84% ROARKE HORSTMEYER: And another interesting thing 00:28:19.600 --> 00:28:23.050 align:middle line:84% is that, so the point spread function 00:28:23.050 --> 00:28:25.300 align:middle line:84% in a camera or the blur spot in a camera, 00:28:25.300 --> 00:28:27.687 align:middle line:84% is given by the shape of the aperture. 00:28:27.687 --> 00:28:29.270 align:middle line:84% So if you have you're taking a picture 00:28:29.270 --> 00:28:32.170 align:middle line:84% and you go out of focus, the things out of focus 00:28:32.170 --> 00:28:37.090 align:middle line:84% have a circle, like a little blur spot. 00:28:37.090 --> 00:28:39.093 align:middle line:84% That's because the aperture is a circle. 00:28:39.093 --> 00:28:40.510 align:middle line:84% With a square aperture, the things 00:28:40.510 --> 00:28:42.225 align:middle line:90% would have a square and so on. 00:28:42.225 --> 00:28:44.350 align:middle line:84% But when you're in focus, even the little blur spot 00:28:44.350 --> 00:28:46.460 align:middle line:84% is given by the shape of the aperture. 00:28:46.460 --> 00:28:49.575 align:middle line:84% So I was putting all these crazy filters in the aperture. 00:28:49.575 --> 00:28:51.700 align:middle line:84% And each filter is going to change the point spread 00:28:51.700 --> 00:28:54.380 align:middle line:84% function depending on the color of the scene. 00:28:54.380 --> 00:28:56.380 align:middle line:84% So basically across my image, I was 00:28:56.380 --> 00:28:58.930 align:middle line:84% getting many different types of point spread function. 00:28:58.930 --> 00:29:03.220 align:middle line:84% Some were strange shapes because of the different properties 00:29:03.220 --> 00:29:05.230 align:middle line:90% of the objects in the scene. 00:29:05.230 --> 00:29:09.760 align:middle line:84% So it's very different problems to try and fix or analyze 00:29:09.760 --> 00:29:16.090 align:middle line:84% because of all the filters in the aperture. 00:29:16.090 --> 00:29:18.760 align:middle line:84% PROFESSOR: So I have just a couple of more points 00:29:18.760 --> 00:29:21.970 align:middle line:90% I want to make before we end. 00:29:21.970 --> 00:29:24.493 align:middle line:84% We've seen right at the start, beginning of the class, 00:29:24.493 --> 00:29:26.410 align:middle line:84% I had these four or five things that we wanted 00:29:26.410 --> 00:29:28.210 align:middle line:90% to improve about cameras. 00:29:28.210 --> 00:29:30.500 align:middle line:84% And one of them was improving the spatial resolution. 00:29:30.500 --> 00:29:33.520 align:middle line:84% So in the case of film it's sort of ambiguous. 00:29:33.520 --> 00:29:35.710 align:middle line:84% Because there is more, it's hard to define 00:29:35.710 --> 00:29:37.220 align:middle line:90% what resolution is for film. 00:29:37.220 --> 00:29:39.640 align:middle line:84% But for sensors or for digital, it's 00:29:39.640 --> 00:29:41.410 align:middle line:90% usually just a number of pixels. 00:29:41.410 --> 00:29:43.780 align:middle line:84% And it would be hard to imagine how 00:29:43.780 --> 00:29:47.620 align:middle line:84% you can increase the information you're getting 00:29:47.620 --> 00:29:48.880 align:middle line:90% from a fixed number of pixels. 00:29:48.880 --> 00:29:50.750 align:middle line:84% But there are techniques for doing that. 00:29:50.750 --> 00:29:55.300 align:middle line:84% And the most popular ones are what's called super resolution. 00:29:55.300 --> 00:29:58.330 align:middle line:84% I'm not going to go too much into the details 00:29:58.330 --> 00:30:00.620 align:middle line:84% of what super resolution is and how you do it. 00:30:00.620 --> 00:30:02.230 align:middle line:84% But just to give you an intuition, 00:30:02.230 --> 00:30:05.230 align:middle line:84% the idea is that you take an image with your camera fixed 00:30:05.230 --> 00:30:08.770 align:middle line:84% at one position and then you move the image sensor 00:30:08.770 --> 00:30:10.990 align:middle line:90% by a fraction of the pixel size. 00:30:10.990 --> 00:30:12.850 align:middle line:90% And then you take another image. 00:30:12.850 --> 00:30:17.080 align:middle line:84% And since you've moved it by a fraction of a pixel size, 00:30:17.080 --> 00:30:18.580 align:middle line:84% the information that you've captured 00:30:18.580 --> 00:30:20.740 align:middle line:84% is actually different from the first one. 00:30:20.740 --> 00:30:23.380 align:middle line:84% And you can combine these two images together 00:30:23.380 --> 00:30:26.110 align:middle line:84% to get a higher resolution image. 00:30:26.110 --> 00:30:28.720 align:middle line:84% There are, obviously, issues with that, one of which 00:30:28.720 --> 00:30:31.690 align:middle line:84% is that you have to move the sensor by less 00:30:31.690 --> 00:30:34.420 align:middle line:84% than a pixel size, which is usually one or two microns. 00:30:34.420 --> 00:30:36.350 align:middle line:84% And it has to be very precisely controlled. 00:30:36.350 --> 00:30:39.130 align:middle line:84% So you find this in really high end camera 00:30:39.130 --> 00:30:41.950 align:middle line:84% that you sign out [INAUDIBLE] cameras which 00:30:41.950 --> 00:30:44.740 align:middle line:84% do this medium format, big medium format cameras. 00:30:44.740 --> 00:30:47.860 align:middle line:84% And there are also fundamental limits to how much 00:30:47.860 --> 00:30:49.240 align:middle line:90% you can do using this. 00:30:49.240 --> 00:30:55.360 align:middle line:84% You can usually not go beyond a 3x or 4x or even just 00:30:55.360 --> 00:30:56.950 align:middle line:90% a 2x increase in resolution. 00:30:56.950 --> 00:30:59.320 align:middle line:84% Anything beyond that is what they call hallucination. 00:30:59.320 --> 00:31:02.590 align:middle line:84% It's you may have create an image which is 100x resolution, 00:31:02.590 --> 00:31:05.500 align:middle line:84% but it doesn't have any information in those higher 00:31:05.500 --> 00:31:06.920 align:middle line:90% frequencies. 00:31:06.920 --> 00:31:08.810 align:middle line:90% So that's super resolution. 00:31:08.810 --> 00:31:11.620 align:middle line:84% So panoramas over time is just taking an image 00:31:11.620 --> 00:31:13.900 align:middle line:84% and just scanning it like this and stitching them 00:31:13.900 --> 00:31:15.820 align:middle line:84% all together, create a large panorama. 00:31:15.820 --> 00:31:19.150 align:middle line:84% You have a big high resolution image and you do so. 00:31:19.150 --> 00:31:21.770 align:middle line:84% And in doing so you also get a wider field of view. 00:31:21.770 --> 00:31:24.370 align:middle line:84% So recently, this technique was adopted 00:31:24.370 --> 00:31:28.930 align:middle line:84% by people at Microsoft Research and the University of Konstanz 00:31:28.930 --> 00:31:34.030 align:middle line:84% where they built a device that essentially scanned 00:31:34.030 --> 00:31:37.130 align:middle line:84% the whole scene and took a whole bunch of images. 00:31:37.130 --> 00:31:40.990 align:middle line:84% So this one was created out of 800 images that were captured 00:31:40.990 --> 00:31:43.180 align:middle line:90% over the period of a few hours. 00:31:43.180 --> 00:31:46.938 align:middle line:84% And then they found correspondence points 00:31:46.938 --> 00:31:48.730 align:middle line:84% and essentially just stitched them together 00:31:48.730 --> 00:31:52.360 align:middle line:84% to create this one large 1.5 megapixel image, I think. 00:31:52.360 --> 00:31:54.700 align:middle line:90% Or 3.5 megapixel image. 00:31:54.700 --> 00:31:59.530 align:middle line:84% And just to give you a sense for how much information is there, 00:31:59.530 --> 00:32:06.300 align:middle line:84% you can actually see the person sitting inside that crane, 00:32:06.300 --> 00:32:07.940 align:middle line:90% whatever that is. 00:32:07.940 --> 00:32:09.530 align:middle line:90% And there is a website. 00:32:09.530 --> 00:32:12.857 align:middle line:90% I think it's called GigaPan. 00:32:12.857 --> 00:32:13.940 align:middle line:90% They sell this device now. 00:32:13.940 --> 00:32:15.860 align:middle line:84% And Microsoft gives you the software. 00:32:15.860 --> 00:32:17.900 align:middle line:84% I think it's called HD View, which 00:32:17.900 --> 00:32:20.270 align:middle line:84% you can use to both view these images 00:32:20.270 --> 00:32:23.618 align:middle line:84% and also generate your own images. 00:32:23.618 --> 00:32:24.410 align:middle line:90% You buy this thing. 00:32:24.410 --> 00:32:25.493 align:middle line:90% You put your camera on it. 00:32:25.493 --> 00:32:28.790 align:middle line:84% It sits there and takes images for a few hours. 00:32:28.790 --> 00:32:31.745 align:middle line:84% And then you just stitch them all together. 00:32:31.745 --> 00:32:36.020 align:middle line:84% There is also a group, there's a husband and wife pro 00:32:36.020 --> 00:32:38.870 align:middle line:84% couple who've been doing this for a while using 00:32:38.870 --> 00:32:39.650 align:middle line:90% film photography. 00:32:39.650 --> 00:32:41.630 align:middle line:84% They have this large format film camera 00:32:41.630 --> 00:32:44.930 align:middle line:84% that they go out and take these huge gigapixel images. 00:32:44.930 --> 00:32:46.670 align:middle line:84% And they have their own custom scanners. 00:32:46.670 --> 00:32:49.820 align:middle line:84% And they will scan them and get a digital image out of it. 00:32:49.820 --> 00:32:52.430 align:middle line:84% And a number of other people who have done this kind of thing. 00:32:52.430 --> 00:32:56.900 align:middle line:84% AUDIENCE: But if this is a consumer device, how 00:32:56.900 --> 00:33:00.770 align:middle line:84% I imagine this actually having the camera move around a lot. 00:33:00.770 --> 00:33:05.010 align:middle line:84% PROFESSOR: So the way they have is something like this. 00:33:05.010 --> 00:33:07.550 align:middle line:84% If you can see it, it just rotates and takes 00:33:07.550 --> 00:33:08.848 align:middle line:90% multiple images. 00:33:08.848 --> 00:33:10.140 align:middle line:90% It doesn't have to move at all. 00:33:10.140 --> 00:33:11.630 align:middle line:90% So it's this thing. 00:33:11.630 --> 00:33:17.330 align:middle line:84% The one that they're selling is it has a very cute shutter 00:33:17.330 --> 00:33:18.200 align:middle line:90% release button also. 00:33:18.200 --> 00:33:20.340 align:middle line:90% So you can just use any camera. 00:33:20.340 --> 00:33:22.893 align:middle line:84% And as long as you can just move the arm 00:33:22.893 --> 00:33:25.310 align:middle line:84% and have it so that it can pretty much [INAUDIBLE] button. 00:33:25.310 --> 00:33:26.690 align:middle line:90% And it will go out. 00:33:26.690 --> 00:33:29.210 align:middle line:84% And you can, I think, program it to say how many images you 00:33:29.210 --> 00:33:30.600 align:middle line:90% want and so on. 00:33:30.600 --> 00:33:32.588 align:middle line:84% So either you can just create a panorama 00:33:32.588 --> 00:33:34.880 align:middle line:84% or you can create a very, very high resolution panorama 00:33:34.880 --> 00:33:35.970 align:middle line:90% like that. 00:33:35.970 --> 00:33:40.400 align:middle line:84% So some of the challenges in this thing that I just 00:33:40.400 --> 00:33:46.370 align:middle line:84% want to point out is there's a huge variation and intensity. 00:33:46.370 --> 00:33:48.830 align:middle line:84% And this is something that we discussed 00:33:48.830 --> 00:33:50.480 align:middle line:90% the high dynamic range issues. 00:33:50.480 --> 00:33:54.650 align:middle line:84% When you're moving the camera and you're doing all of this, 00:33:54.650 --> 00:33:57.320 align:middle line:84% you could have parts of the image that are much, much, much 00:33:57.320 --> 00:33:59.750 align:middle line:84% darker than other parts or much brighter. 00:33:59.750 --> 00:34:01.550 align:middle line:84% And then how you stitch them together, 00:34:01.550 --> 00:34:05.240 align:middle line:84% what kind of exposure issues do you have to take care of? 00:34:05.240 --> 00:34:07.790 align:middle line:84% Because you just do auto exposure then 00:34:07.790 --> 00:34:11.520 align:middle line:84% one part of the image may be too bright than the other. 00:34:11.520 --> 00:34:13.429 align:middle line:90% And it might not line up. 00:34:13.429 --> 00:34:15.080 align:middle line:84% You might not get the alignment right 00:34:15.080 --> 00:34:17.600 align:middle line:84% or you might have issues with finding correspondences. 00:34:17.600 --> 00:34:21.469 align:middle line:84% And all of that is taken care of the software from Microsoft 00:34:21.469 --> 00:34:22.389 align:middle line:90% called HD View. 00:34:22.389 --> 00:34:25.739 align:middle line:90% 00:34:25.739 --> 00:34:27.960 align:middle line:84% Another way of doing high resolution, 00:34:27.960 --> 00:34:30.870 align:middle line:84% this is again going back to those three basic things. 00:34:30.870 --> 00:34:34.770 align:middle line:84% This one I just talked about this was epsilon in time. 00:34:34.770 --> 00:34:36.750 align:middle line:84% But you could do epsilon in sensors. 00:34:36.750 --> 00:34:39.420 align:middle line:84% You can have 500 cameras and you just 00:34:39.420 --> 00:34:41.620 align:middle line:84% take lots of images at the same time. 00:34:41.620 --> 00:34:45.460 align:middle line:84% So this is similar to the camera array we saw earlier. 00:34:45.460 --> 00:34:50.190 align:middle line:84% But the difference in this is that the cameras are all 00:34:50.190 --> 00:34:53.741 align:middle line:84% looking out parallel to one another. 00:34:53.741 --> 00:34:55.199 align:middle line:84% In the previous case, they were all 00:34:55.199 --> 00:34:57.490 align:middle line:90% looking at something in between. 00:34:57.490 --> 00:34:59.040 align:middle line:90% So there was a lot of overlap. 00:34:59.040 --> 00:35:02.293 align:middle line:84% In this, the overlap is much less. 00:35:02.293 --> 00:35:04.710 align:middle line:84% It says for people-- and I think it's even less than that. 00:35:04.710 --> 00:35:07.395 align:middle line:84% And what you can do with this is you 00:35:07.395 --> 00:35:09.660 align:middle line:84% can get all of these images and then, 00:35:09.660 --> 00:35:11.370 align:middle line:84% just again, find the correspondences 00:35:11.370 --> 00:35:14.760 align:middle line:84% and stitch them all together and get something like this. 00:35:14.760 --> 00:35:16.650 align:middle line:84% This is a very, very high resolution, 00:35:16.650 --> 00:35:20.382 align:middle line:90% and it has lots of information. 00:35:20.382 --> 00:35:21.840 align:middle line:84% And then again, you can again, find 00:35:21.840 --> 00:35:25.020 align:middle line:84% those correspondences, like them and just stitch them together. 00:35:25.020 --> 00:35:29.040 align:middle line:84% But there's also issues of geometric and color calibration 00:35:29.040 --> 00:35:33.570 align:middle line:84% and also high dynamic range issues, which I briefly 00:35:33.570 --> 00:35:34.450 align:middle line:90% discussed earlier. 00:35:34.450 --> 00:35:37.050 align:middle line:84% So this is what you get once you fix all of those. 00:35:37.050 --> 00:35:39.880 align:middle line:84% For some reason, it's really dark here. 00:35:39.880 --> 00:35:42.510 align:middle line:84% And again, it shows you can really zoom in and get 00:35:42.510 --> 00:35:43.620 align:middle line:90% a high resolution image. 00:35:43.620 --> 00:35:45.990 align:middle line:84% I don't think this is anywhere close to [INAUDIBLE] 00:35:45.990 --> 00:35:50.010 align:middle line:84% Because this is a much older project. 00:35:50.010 --> 00:35:53.160 align:middle line:84% And again similar to the assorted pixels, 00:35:53.160 --> 00:35:55.140 align:middle line:84% they actually have different exposures 00:35:55.140 --> 00:35:56.190 align:middle line:90% for different cameras. 00:35:56.190 --> 00:35:58.170 align:middle line:84% And so you can combine all of that information 00:35:58.170 --> 00:36:00.263 align:middle line:84% together to get not just a high resolution 00:36:00.263 --> 00:36:02.430 align:middle line:84% image but also a high resolution, high dynamic range 00:36:02.430 --> 00:36:02.930 align:middle line:90% image. 00:36:02.930 --> 00:36:06.120 align:middle line:90% 00:36:06.120 --> 00:36:07.800 align:middle line:84% The last thing I want to talk about 00:36:07.800 --> 00:36:10.920 align:middle line:84% is increasing the temporal resolution. 00:36:10.920 --> 00:36:13.110 align:middle line:84% So far, we've seen how we can increase 00:36:13.110 --> 00:36:14.250 align:middle line:90% the spatial resolution. 00:36:14.250 --> 00:36:19.530 align:middle line:84% We've seen how we can increase the dynamic range and focus 00:36:19.530 --> 00:36:23.610 align:middle line:84% depth of [INAUDIBLE] and the other thing that remains, 00:36:23.610 --> 00:36:24.630 align:middle line:90% the temporal resolution. 00:36:24.630 --> 00:36:28.140 align:middle line:84% And you can now buy cameras that have 1,000 frames per second 00:36:28.140 --> 00:36:28.960 align:middle line:90% or even more. 00:36:28.960 --> 00:36:30.960 align:middle line:84% And you can just hit the shutter and it 00:36:30.960 --> 00:36:32.970 align:middle line:90% takes a whole series of images. 00:36:32.970 --> 00:36:36.630 align:middle line:84% But this is from a few years ago where what they said 00:36:36.630 --> 00:36:39.690 align:middle line:84% was that instead of taking one camera with very high frame 00:36:39.690 --> 00:36:42.990 align:middle line:84% rate, what if you took multiple cameras with more 00:36:42.990 --> 00:36:45.870 align:middle line:84% reasonable frame rates and then you combine that information 00:36:45.870 --> 00:36:47.040 align:middle line:90% together. 00:36:47.040 --> 00:36:50.790 align:middle line:84% And sure enough, they came up with their own camera array. 00:36:50.790 --> 00:36:52.140 align:middle line:90% This is also from Stanford. 00:36:52.140 --> 00:36:58.200 align:middle line:84% And you can combine, I think they triggered them in a way 00:36:58.200 --> 00:37:02.440 align:middle line:84% that they can then later combine the information. 00:37:02.440 --> 00:37:05.010 align:middle line:84% But again you need to calibrate the images 00:37:05.010 --> 00:37:10.440 align:middle line:84% and also color correct them and so on. 00:37:10.440 --> 00:37:14.470 align:middle line:90% It's just a [INAUDIBLE]. 00:37:14.470 --> 00:37:17.120 align:middle line:90% 00:37:17.120 --> 00:37:21.800 align:middle line:84% You can see this off like artifacts as the thing moves 00:37:21.800 --> 00:37:25.220 align:middle line:84% and because the calibration is imperfect. 00:37:25.220 --> 00:37:27.410 align:middle line:84% And each of these is a relatively low resolution 00:37:27.410 --> 00:37:29.553 align:middle line:84% camera, I think about six [INAUDIBLE].. 00:37:29.553 --> 00:37:33.252 align:middle line:90% 00:37:33.252 --> 00:37:34.210 align:middle line:90% So that's basically it. 00:37:34.210 --> 00:37:36.000 align:middle line:90% So that's epsilon photography. 00:37:36.000 --> 00:37:40.530 align:middle line:84% It's how we enhance film-based photography. 00:37:40.530 --> 00:37:44.340 align:middle line:84% And the idea is to modify the exposure settings, spectrum 00:37:44.340 --> 00:37:47.040 align:middle line:84% of color, focus, camera scene, illumination, 00:37:47.040 --> 00:37:50.530 align:middle line:84% basically anything else that is a parameter of the camera 00:37:50.530 --> 00:37:53.940 align:middle line:84% and take multiple images or take images [INAUDIBLE] 00:37:53.940 --> 00:37:56.190 align:middle line:90% over time or sensor or pixels. 00:37:56.190 --> 00:37:59.650 align:middle line:84% And the end result is get a better camera. 00:37:59.650 --> 00:38:02.850 align:middle line:84% And as we'll see in the next class and future classes 00:38:02.850 --> 00:38:06.150 align:middle line:84% that this is not the interesting part of computational cameras. 00:38:06.150 --> 00:38:10.210 align:middle line:84% This is just something that it's good to know about. 00:38:10.210 --> 00:38:12.930 align:middle line:84% And there's still research going on on many of these topics. 00:38:12.930 --> 00:38:15.720 align:middle line:84% It's just, it's furthering or making 00:38:15.720 --> 00:38:18.480 align:middle line:84% even better cameras the way cameras were 00:38:18.480 --> 00:38:22.110 align:middle line:84% for more than a century rather than coming up with cameras 00:38:22.110 --> 00:38:23.900 align:middle line:90% that do new things. 00:38:23.900 --> 00:38:26.120 align:middle line:90% I guess that's it. 00:38:26.120 --> 00:38:27.000 align:middle line:90%