WEBVTT 00:00:00.000 --> 00:00:02.490 align:middle line:84% The following content is provided under a Creative 00:00:02.490 --> 00:00:03.730 align:middle line:90% Commons license. 00:00:03.730 --> 00:00:06.120 align:middle line:84% Your support will help MIT OpenCourseWare 00:00:06.120 --> 00:00:10.090 align:middle line:84% continue to offer high quality educational resources for free. 00:00:10.090 --> 00:00:12.720 align:middle line:84% To make a donation or to view additional materials 00:00:12.720 --> 00:00:16.650 align:middle line:84% from hundreds of MIT courses, visit MIT OpenCourseWare 00:00:16.650 --> 00:00:18.850 align:middle line:90% at ocw.mit.edu. 00:00:18.850 --> 00:00:21.345 align:middle line:90% 00:00:21.345 --> 00:00:25.875 align:middle line:84% PROFESSOR: So let me hand it over to Ankit now. 00:00:25.875 --> 00:00:27.375 align:middle line:84% And we're going to talk about color. 00:00:27.375 --> 00:00:30.627 align:middle line:90% 00:00:30.627 --> 00:00:32.960 align:middle line:84% ANKIT MOHAN: So I'm going to start with something that I 00:00:32.960 --> 00:00:35.960 align:middle line:84% discussed a few weeks back when we were talking about epsilon 00:00:35.960 --> 00:00:40.670 align:middle line:84% photography, and if you remember the slide I had back then. 00:00:40.670 --> 00:00:43.400 align:middle line:84% And this is one of the oldest ways of creating color images. 00:00:43.400 --> 00:00:45.290 align:middle line:84% You just capture the images over time, 00:00:45.290 --> 00:00:47.960 align:middle line:84% one with a green filter in front of the camera lens, 00:00:47.960 --> 00:00:50.030 align:middle line:84% one with a red one, with a blue filter. 00:00:50.030 --> 00:00:53.060 align:middle line:84% And from these three images, if you project them back 00:00:53.060 --> 00:00:56.157 align:middle line:84% on a white surface, again, through three 00:00:56.157 --> 00:00:58.490 align:middle line:84% different projectors-- one with a green, one with a red, 00:00:58.490 --> 00:01:00.240 align:middle line:84% and one with a blue filter in front of it, 00:01:00.240 --> 00:01:03.590 align:middle line:84% you will get what looks like a color image to the human eye. 00:01:03.590 --> 00:01:06.680 align:middle line:84% And this is stuff that was done more than a century ago. 00:01:06.680 --> 00:01:09.740 align:middle line:84% And you can take these images now, 00:01:09.740 --> 00:01:11.660 align:middle line:84% these individual grayscale images, 00:01:11.660 --> 00:01:13.160 align:middle line:84% and color tint them and then again, 00:01:13.160 --> 00:01:15.147 align:middle line:84% add them back together, similar to what 00:01:15.147 --> 00:01:17.730 align:middle line:84% you did for your assignment when you were adding the lighting. 00:01:17.730 --> 00:01:19.460 align:middle line:84% And you can create these digitally. 00:01:19.460 --> 00:01:22.100 align:middle line:84% And now there's a whole library of-- 00:01:22.100 --> 00:01:25.340 align:middle line:84% Library of Congress has this huge database 00:01:25.340 --> 00:01:27.530 align:middle line:84% of images that they're, by registering them, 00:01:27.530 --> 00:01:29.600 align:middle line:84% they're recreating these color images 00:01:29.600 --> 00:01:32.130 align:middle line:90% from black and white images. 00:01:32.130 --> 00:01:36.530 align:middle line:84% So this is one way of creating color in photography. 00:01:36.530 --> 00:01:40.953 align:middle line:84% Another related similar thing to this is the color wheel. 00:01:40.953 --> 00:01:43.370 align:middle line:84% I just wanted to-- this is something that, again, I talked 00:01:43.370 --> 00:01:45.110 align:middle line:90% about in the previous class. 00:01:45.110 --> 00:01:47.368 align:middle line:84% And it's something that's used more for projectors. 00:01:47.368 --> 00:01:49.160 align:middle line:84% You have this color wheel rotating in front 00:01:49.160 --> 00:01:52.370 align:middle line:84% of this-- probably has one if it's a DLP projector. 00:01:52.370 --> 00:01:55.820 align:middle line:84% And at any given time, you're just 00:01:55.820 --> 00:01:57.980 align:middle line:84% projecting one component of the light. 00:01:57.980 --> 00:02:00.170 align:middle line:84% You're either green, red, or blue. 00:02:00.170 --> 00:02:03.290 align:middle line:84% And the human eye actually integrates it over time. 00:02:03.290 --> 00:02:06.757 align:middle line:84% Because it happens in rapid succession one after the other. 00:02:06.757 --> 00:02:08.840 align:middle line:84% So it's a little different from the previous case, 00:02:08.840 --> 00:02:11.900 align:middle line:84% in which you were projecting all three at the same time 00:02:11.900 --> 00:02:14.270 align:middle line:90% and light was adding in space. 00:02:14.270 --> 00:02:16.700 align:middle line:84% Here, light gets integrated over time. 00:02:16.700 --> 00:02:20.000 align:middle line:90% You get the effect of color. 00:02:20.000 --> 00:02:23.150 align:middle line:84% So a third thing that I discussed previously 00:02:23.150 --> 00:02:26.360 align:middle line:84% in the previous task force, this concept of three CCD sensors, 00:02:26.360 --> 00:02:30.470 align:middle line:84% where you use these dichroic prisms and mirrors and you 00:02:30.470 --> 00:02:33.590 align:middle line:84% split light into the three wavelength regions. 00:02:33.590 --> 00:02:36.650 align:middle line:84% And then you have three separate sensors that 00:02:36.650 --> 00:02:38.000 align:middle line:90% capture the three wavelengths. 00:02:38.000 --> 00:02:41.210 align:middle line:84% And you get, using three monochrome CCDs, 00:02:41.210 --> 00:02:42.575 align:middle line:90% you can capture a color image. 00:02:42.575 --> 00:02:45.510 align:middle line:90% 00:02:45.510 --> 00:02:48.690 align:middle line:84% This is probably what's the most popular in most still 00:02:48.690 --> 00:02:51.330 align:middle line:84% cameras, and also many of the digital cameras. 00:02:51.330 --> 00:02:54.270 align:middle line:84% And it's basically what's called the Bear mosaic. 00:02:54.270 --> 00:02:57.180 align:middle line:84% That's because the person who invented it from Kodak, 00:02:57.180 --> 00:02:58.530 align:middle line:90% his name was Bear. 00:02:58.530 --> 00:03:01.860 align:middle line:84% And he had this patented in, I think, mid 00:03:01.860 --> 00:03:04.560 align:middle line:84% or late '70s, which describes this technique. 00:03:04.560 --> 00:03:07.260 align:middle line:84% The basic idea is that instead of having monochrome pixels, 00:03:07.260 --> 00:03:10.650 align:middle line:84% you put these little filters, tiny filters, 00:03:10.650 --> 00:03:11.940 align:middle line:90% on top of the pixels. 00:03:11.940 --> 00:03:17.520 align:middle line:84% And the filters, you have two green filters, and one red, 00:03:17.520 --> 00:03:19.680 align:middle line:84% and one blue filter in a 2 by 2 region. 00:03:19.680 --> 00:03:23.100 align:middle line:84% And then this 2 by 2 thing is tiled all over the sensor. 00:03:23.100 --> 00:03:28.510 align:middle line:84% So any one pixel is only sensing one wavelength, either red, 00:03:28.510 --> 00:03:30.270 align:middle line:90% green, or blue. 00:03:30.270 --> 00:03:31.980 align:middle line:84% When I say wavelength, it's actually 00:03:31.980 --> 00:03:33.660 align:middle line:90% a whole band of wavelengths. 00:03:33.660 --> 00:03:36.670 align:middle line:84% But let's just stick with color for now. 00:03:36.670 --> 00:03:39.270 align:middle line:84% And so it's sensing just one of these three colors. 00:03:39.270 --> 00:03:43.290 align:middle line:84% And then they use clever demosaicing algorithms, 00:03:43.290 --> 00:03:47.530 align:middle line:84% which essentially interpolate between the image that sense. 00:03:47.530 --> 00:03:51.360 align:middle line:84% So for example, at a pixel with sensors only red, 00:03:51.360 --> 00:03:53.910 align:middle line:84% you would interpolate between the neighboring green pixels 00:03:53.910 --> 00:03:57.330 align:middle line:84% in order to estimate what the green color is over there, 00:03:57.330 --> 00:04:00.900 align:middle line:84% while still making use of the color, that center 00:04:00.900 --> 00:04:01.960 align:middle line:90% of the red color. 00:04:01.960 --> 00:04:05.740 align:middle line:84% So what you see in most images is almost kind of you're 00:04:05.740 --> 00:04:07.230 align:middle line:84% hallucinating the higher resolution 00:04:07.230 --> 00:04:10.560 align:middle line:84% because you are going up by a factor of 3 00:04:10.560 --> 00:04:12.630 align:middle line:84% in the resolution in the image when you 00:04:12.630 --> 00:04:14.610 align:middle line:90% do this kind of interpolation. 00:04:14.610 --> 00:04:17.829 align:middle line:84% PROFESSOR: And so a 4 megapixel camera, 00:04:17.829 --> 00:04:21.300 align:middle line:84% and here it looks like we have for every four 00:04:21.300 --> 00:04:27.190 align:middle line:84% pixels we have two green, one blue, and one red. 00:04:27.190 --> 00:04:29.700 align:middle line:84% So if we take a 4 megapixel picture, 00:04:29.700 --> 00:04:33.945 align:middle line:84% how many pixels are actually taking each of those colors? 00:04:33.945 --> 00:04:37.940 align:middle line:90% 00:04:37.940 --> 00:04:40.010 align:middle line:84% This four megapixel camera actually giving you 00:04:40.010 --> 00:04:43.160 align:middle line:84% 12 megapixels, because when image comes out, 00:04:43.160 --> 00:04:45.500 align:middle line:84% it's four megapixel in red, four megapixel in blue, 00:04:45.500 --> 00:04:46.730 align:middle line:90% and four megapixel in green. 00:04:46.730 --> 00:04:50.600 align:middle line:90% 00:04:50.600 --> 00:04:51.970 align:middle line:90% So what is going on? 00:04:51.970 --> 00:04:54.220 align:middle line:84% STUDENT: Well, I mean, you get the resolution for each 00:04:54.220 --> 00:04:56.428 align:middle line:84% of those colors, since they're separated in the space 00:04:56.428 --> 00:04:58.910 align:middle line:84% by the other colors being in space, as well. 00:04:58.910 --> 00:05:01.160 align:middle line:84% PROFESSOR: So in this 4 megapixel, 00:05:01.160 --> 00:05:02.970 align:middle line:84% there are only 4 million pixels total. 00:05:02.970 --> 00:05:03.845 align:middle line:90% STUDENT: Yeah, total. 00:05:03.845 --> 00:05:04.345 align:middle line:90% [INAUDIBLE] 00:05:04.345 --> 00:05:05.960 align:middle line:84% PROFESSOR: And then two megapixels 00:05:05.960 --> 00:05:10.320 align:middle line:84% are green-- one megapixel is red, one megapixel is blue. 00:05:10.320 --> 00:05:13.460 align:middle line:84% And so what's the benefit of, if you can pair this 00:05:13.460 --> 00:05:19.070 align:middle line:84% with a previous design, where you had three separate CCDs, 00:05:19.070 --> 00:05:21.740 align:middle line:90% one for each color. 00:05:21.740 --> 00:05:23.858 align:middle line:84% What's the benefit, and what's the disadvantage? 00:05:23.858 --> 00:05:25.490 align:middle line:90% STUDENT: Money. 00:05:25.490 --> 00:05:26.090 align:middle line:90% Cheaper. 00:05:26.090 --> 00:05:26.660 align:middle line:90% It's cheaper. 00:05:26.660 --> 00:05:27.170 align:middle line:90% Yeah. 00:05:27.170 --> 00:05:28.378 align:middle line:90% PROFESSOR: Why is it cheaper. 00:05:28.378 --> 00:05:30.380 align:middle line:90% STUDENT: One CCD. 00:05:30.380 --> 00:05:32.900 align:middle line:90% STUDENT: And it's less optics. 00:05:32.900 --> 00:05:35.950 align:middle line:90% PROFESSOR: Less optics climate. 00:05:35.950 --> 00:05:37.750 align:middle line:90% Alignment is easy. 00:05:37.750 --> 00:05:39.370 align:middle line:90% What's the disadvantage? 00:05:39.370 --> 00:05:41.275 align:middle line:84% STUDENT: Plus you're blocking light. 00:05:41.275 --> 00:05:43.760 align:middle line:84% PROFESSOR: You're getting-- effectively 00:05:43.760 --> 00:05:46.570 align:middle line:84% getting one third light because every time you 00:05:46.570 --> 00:05:50.890 align:middle line:84% sense one color, the light for other two colors 00:05:50.890 --> 00:05:52.840 align:middle line:90% is being thrown out. 00:05:52.840 --> 00:05:55.210 align:middle line:84% And when Mike talks about a little bit later 00:05:55.210 --> 00:05:59.110 align:middle line:84% about some other multispectral cameras, 00:05:59.110 --> 00:06:02.140 align:middle line:84% this will become the biggest issue of the notion 00:06:02.140 --> 00:06:04.588 align:middle line:90% of [INAUDIBLE] light. 00:06:04.588 --> 00:06:08.320 align:middle line:84% STUDENT: And also can you perfectly reconstruct 00:06:08.320 --> 00:06:09.860 align:middle line:90% the image. 00:06:09.860 --> 00:06:11.600 align:middle line:84% PROFESSOR: So this one was an issue. 00:06:11.600 --> 00:06:15.910 align:middle line:84% So, most of the reconstruction, the interpolation that we just 00:06:15.910 --> 00:06:20.050 align:middle line:84% talked about is based on some assumption 00:06:20.050 --> 00:06:21.970 align:middle line:90% about the natural scene. 00:06:21.970 --> 00:06:24.040 align:middle line:84% But if your scene is not natural, 00:06:24.040 --> 00:06:27.700 align:middle line:84% maybe it's a black and white text 00:06:27.700 --> 00:06:32.390 align:middle line:84% or you have very fine stripes on your shirt, in that case, 00:06:32.390 --> 00:06:35.375 align:middle line:84% the interpolation would not give you the right results. 00:06:35.375 --> 00:06:36.250 align:middle line:90% And you see it often. 00:06:36.250 --> 00:06:38.620 align:middle line:84% You can see this strange [INAUDIBLE].. 00:06:38.620 --> 00:06:39.370 align:middle line:90% ANKIT MOHAN: Yeah. 00:06:39.370 --> 00:06:43.090 align:middle line:84% So, one of the things they do in order 00:06:43.090 --> 00:06:45.550 align:middle line:84% to avoid this kind of thing is that they place a low pass 00:06:45.550 --> 00:06:47.830 align:middle line:84% filter immediately on top of the sensor 00:06:47.830 --> 00:06:51.340 align:middle line:84% so that you get rid of any such high frequency 00:06:51.340 --> 00:06:54.760 align:middle line:90% details in a single color. 00:06:54.760 --> 00:06:56.020 align:middle line:90% But still, you get-- 00:06:56.020 --> 00:06:58.720 align:middle line:84% I mean, in many of these images, you do see color [INAUDIBLE] 00:06:58.720 --> 00:07:02.620 align:middle line:84% so you have some weird artifacts in one color channel 00:07:02.620 --> 00:07:06.050 align:middle line:84% and not in the others or rainbow effect. 00:07:06.050 --> 00:07:08.780 align:middle line:84% And that shows up, especially along near the edges, 00:07:08.780 --> 00:07:11.070 align:middle line:90% and so on. 00:07:11.070 --> 00:07:12.840 align:middle line:84% Someone-- did someone have a question? 00:07:12.840 --> 00:07:16.030 align:middle line:90% 00:07:16.030 --> 00:07:18.470 align:middle line:84% So, one of the things as Ramesh mentioned, 00:07:18.470 --> 00:07:22.220 align:middle line:84% is that you are throwing away 1/3 or 2/3 of the light 00:07:22.220 --> 00:07:23.520 align:middle line:90% in doing the previous one. 00:07:23.520 --> 00:07:26.912 align:middle line:84% So, there is this recent pattern that Nikon recently 00:07:26.912 --> 00:07:28.370 align:middle line:84% introduced where what they're doing 00:07:28.370 --> 00:07:31.640 align:middle line:84% is combining these two notions that we just discussed. 00:07:31.640 --> 00:07:35.960 align:middle line:84% They have these red, green, and blue sensors 00:07:35.960 --> 00:07:37.320 align:middle line:90% at the pixel level. 00:07:37.320 --> 00:07:39.830 align:middle line:84% But they-- instead of using three separate filters, 00:07:39.830 --> 00:07:41.540 align:middle line:84% they use dichroic mirrors in order 00:07:41.540 --> 00:07:43.970 align:middle line:84% to separate the light that's falling on the sensor 00:07:43.970 --> 00:07:46.440 align:middle line:90% into the R, G and B component. 00:07:46.440 --> 00:07:49.850 align:middle line:84% So, it's similar to the first three CCD case 00:07:49.850 --> 00:07:51.350 align:middle line:84% that they're using dichroic mirrors. 00:07:51.350 --> 00:07:53.340 align:middle line:84% They're not using any light, but it's all 00:07:53.340 --> 00:07:54.590 align:middle line:90% happening at the sensor level. 00:07:54.590 --> 00:07:58.340 align:middle line:84% So, conceivably they can build this thing in the semiconductor 00:07:58.340 --> 00:07:58.890 align:middle line:90% itself. 00:07:58.890 --> 00:08:02.470 align:middle line:84% And so it's much cheaper than using prisms outside of one. 00:08:02.470 --> 00:08:05.360 align:middle line:90% 00:08:05.360 --> 00:08:11.030 align:middle line:84% PROFESSOR: Anybody familiar with dichroic materials? 00:08:11.030 --> 00:08:13.490 align:middle line:84% It's basically a type of [INAUDIBLE] in simplest words. 00:08:13.490 --> 00:08:19.640 align:middle line:84% It's a type of a glass where at the right angle, 00:08:19.640 --> 00:08:22.610 align:middle line:84% if you shine light at the right angle 00:08:22.610 --> 00:08:25.730 align:middle line:84% particular wavelength will pass through and all of the elements 00:08:25.730 --> 00:08:27.147 align:middle line:90% will be reflected. 00:08:27.147 --> 00:08:28.730 align:middle line:84% ANKIT MOHAN: It's essentially, I think 00:08:28.730 --> 00:08:31.280 align:middle line:84% it's total internal reflection is what it makes use of. 00:08:31.280 --> 00:08:34.610 align:middle line:84% And the threshold is different for different wavelengths 00:08:34.610 --> 00:08:37.760 align:middle line:84% when the light is going to get total internal reflected 00:08:37.760 --> 00:08:40.340 align:middle line:84% or just keep going straight into the next material. 00:08:40.340 --> 00:08:42.620 align:middle line:84% I'm using a series of such mirrors 00:08:42.620 --> 00:08:45.200 align:middle line:84% they can separate the three weapons. 00:08:45.200 --> 00:08:47.777 align:middle line:84% PROFESSOR: And in the break, we'll have the soap bubbles 00:08:47.777 --> 00:08:49.610 align:middle line:84% and then we'll demonstrate the same concept. 00:08:49.610 --> 00:08:53.710 align:middle line:90% 00:08:53.710 --> 00:08:56.080 align:middle line:84% ANKIT MOHAN: So, one other popular-- 00:08:56.080 --> 00:08:58.810 align:middle line:84% semi-popular sensor design is what's 00:08:58.810 --> 00:09:00.670 align:middle line:90% called the Foveon X3 sensor. 00:09:00.670 --> 00:09:02.950 align:middle line:84% And instead of using three separate pixels, 00:09:02.950 --> 00:09:05.260 align:middle line:84% each pixel having a different filter 00:09:05.260 --> 00:09:07.420 align:middle line:84% on top of that, the design of this sensor 00:09:07.420 --> 00:09:10.180 align:middle line:84% is very similar to that, which is used in film. 00:09:10.180 --> 00:09:11.890 align:middle line:84% So there-- in film, they actually 00:09:11.890 --> 00:09:16.270 align:middle line:84% have three separate emulsions or three or four separate layers 00:09:16.270 --> 00:09:20.110 align:middle line:84% of emulsions each sensitive to a different wavelength of light. 00:09:20.110 --> 00:09:22.060 align:middle line:84% And similarly over here, they have 00:09:22.060 --> 00:09:25.120 align:middle line:84% as you keep going down deeper into the pixel, 00:09:25.120 --> 00:09:27.550 align:middle line:84% different depths of the pixel actually 00:09:27.550 --> 00:09:29.930 align:middle line:84% are sensitive to different wavelengths of light. 00:09:29.930 --> 00:09:31.630 align:middle line:84% So, the top region is sensitive, more 00:09:31.630 --> 00:09:36.130 align:middle line:84% sensitive to blue and the next part, the green, and the bottom 00:09:36.130 --> 00:09:36.970 align:middle line:90% most is to red. 00:09:36.970 --> 00:09:39.730 align:middle line:84% And so a single pixel can actually 00:09:39.730 --> 00:09:42.430 align:middle line:84% sense all three colors, all three wavelengths, 00:09:42.430 --> 00:09:44.120 align:middle line:90% as they're falling down on it. 00:09:44.120 --> 00:09:46.780 align:middle line:84% And the advantage is that you don't 00:09:46.780 --> 00:09:50.300 align:middle line:84% need to put this low pass filter on top of it 00:09:50.300 --> 00:09:52.330 align:middle line:84% because it's traveling the spatial-- 00:09:52.330 --> 00:09:55.040 align:middle line:84% multiplexing-- you're doing this multiplexing in depth. 00:09:55.040 --> 00:09:57.160 align:middle line:90% So, that's a big advantage. 00:09:57.160 --> 00:10:00.250 align:middle line:84% In terms of the resolution, yes, you do get-- 00:10:00.250 --> 00:10:02.860 align:middle line:84% you don't lose the resolution like do 00:10:02.860 --> 00:10:04.870 align:middle line:90% in the case of a Bayer filter. 00:10:04.870 --> 00:10:08.590 align:middle line:84% But so far, when they caught the number of 00:10:08.590 --> 00:10:11.050 align:middle line:84% pixels that they simply multiplied by 3. 00:10:11.050 --> 00:10:13.332 align:middle line:84% So, when sigma is the manufacturer, 00:10:13.332 --> 00:10:15.790 align:middle line:84% that manufactures these Foveon sensors now they [INAUDIBLE] 00:10:15.790 --> 00:10:16.460 align:middle line:90% over Foveon. 00:10:16.460 --> 00:10:18.760 align:middle line:84% So, when they say they have a 12 megapixel sensor, 00:10:18.760 --> 00:10:20.800 align:middle line:84% it's really a four megapixel sensor. 00:10:20.800 --> 00:10:23.020 align:middle line:84% They're just call it a 12 megapixel sensor 00:10:23.020 --> 00:10:25.490 align:middle line:84% because there are three elements for each pixel. 00:10:25.490 --> 00:10:28.420 align:middle line:84% So, it's not clear whether that's really-- 00:10:28.420 --> 00:10:30.310 align:middle line:84% again, in terms of the resolution. 00:10:30.310 --> 00:10:33.460 align:middle line:84% But a definite gain is that you don't need a low pass 00:10:33.460 --> 00:10:36.160 align:middle line:84% filter over it so you don't get these-- 00:10:36.160 --> 00:10:39.190 align:middle line:84% you get-- you're able to capture much higher frequency 00:10:39.190 --> 00:10:39.790 align:middle line:90% information. 00:10:39.790 --> 00:10:41.457 align:middle line:84% PROFESSOR: So, you can't win either way. 00:10:41.457 --> 00:10:43.930 align:middle line:84% When they say it's four megapixel traditional camera 00:10:43.930 --> 00:10:46.408 align:middle line:84% makers, you only get a two megapixel green image. 00:10:46.408 --> 00:10:47.200 align:middle line:90% ANKIT MOHAN: Right. 00:10:47.200 --> 00:10:50.208 align:middle line:84% PROFESSOR: And when they say it's 12 megapixel, 00:10:50.208 --> 00:10:52.000 align:middle line:84% you still get a four megapixel green image. 00:10:52.000 --> 00:10:53.020 align:middle line:90% ANKIT MOHAN: Right. 00:10:53.020 --> 00:10:54.760 align:middle line:84% But one other disadvantage of this 00:10:54.760 --> 00:10:57.640 align:middle line:84% is that unlike the previous case where you can almost 00:10:57.640 --> 00:10:59.620 align:middle line:84% arbitrarily choose what filter you want 00:10:59.620 --> 00:11:04.240 align:middle line:84% over each pixel in this case, the separation 00:11:04.240 --> 00:11:07.120 align:middle line:84% between the red, green, and blue channels is not as great. 00:11:07.120 --> 00:11:09.830 align:middle line:84% You don't-- they apparently don't have as much control. 00:11:09.830 --> 00:11:12.040 align:middle line:84% So, they need to do a lot of software processing 00:11:12.040 --> 00:11:15.560 align:middle line:84% on the image that's captured in order to separate 00:11:15.560 --> 00:11:16.810 align:middle line:90% the red, green, and blue ones. 00:11:16.810 --> 00:11:19.810 align:middle line:84% Possibly, they're doing some sort of patchworks 00:11:19.810 --> 00:11:22.060 align:middle line:84% based on some of image prior, so you 00:11:22.060 --> 00:11:25.060 align:middle line:84% would have some image artifacts over there which might not 00:11:25.060 --> 00:11:26.680 align:middle line:84% be these moire artifacts, but you'll 00:11:26.680 --> 00:11:29.260 align:middle line:90% have some color artifacts. 00:11:29.260 --> 00:11:32.620 align:middle line:84% PROFESSOR: And the actual profile is very misleading 00:11:32.620 --> 00:11:37.790 align:middle line:84% because when you see this diagram on the left, 00:11:37.790 --> 00:11:40.128 align:middle line:84% the blue is basically getting everything. 00:11:40.128 --> 00:11:41.170 align:middle line:90% So, it's something like-- 00:11:41.170 --> 00:11:44.850 align:middle line:90% 00:11:44.850 --> 00:11:47.580 align:middle line:84% this is your blue, green, and red. 00:11:47.580 --> 00:11:51.480 align:middle line:84% Then the blue pixel is getting almost everything 00:11:51.480 --> 00:11:54.843 align:middle line:84% and the green pixel is getting a little bit less 00:11:54.843 --> 00:11:56.760 align:middle line:84% and the red pixel is getting whatever remains. 00:11:56.760 --> 00:11:59.790 align:middle line:90% 00:11:59.790 --> 00:12:03.060 align:middle line:84% So, it's not-- it's not like blue, green, and red. 00:12:03.060 --> 00:12:06.090 align:middle line:84% So, the picture on the right is completely misleading. 00:12:06.090 --> 00:12:10.048 align:middle line:84% It's getting some free values that are highly overlapped. 00:12:10.048 --> 00:12:12.090 align:middle line:84% And from that, they're going to do some inversion 00:12:12.090 --> 00:12:14.460 align:middle line:90% and figure it out actually. 00:12:14.460 --> 00:12:17.910 align:middle line:84% So in the very simplest words we can think of, 00:12:17.910 --> 00:12:21.720 align:middle line:84% the first one here is everything. 00:12:21.720 --> 00:12:26.110 align:middle line:84% The second one here is just this and the last one gets this. 00:12:26.110 --> 00:12:29.970 align:middle line:84% And from that, they can figure out what red and blue is. 00:12:29.970 --> 00:12:33.560 align:middle line:90% 00:12:33.560 --> 00:12:37.730 align:middle line:84% ANKIT MOHAN: So, I should also say that there's nothing really 00:12:37.730 --> 00:12:39.740 align:middle line:90% holy about this RGB design. 00:12:39.740 --> 00:12:42.530 align:middle line:84% It's just that this was the one that was proposed first. 00:12:42.530 --> 00:12:44.450 align:middle line:90% And this is what's being used. 00:12:44.450 --> 00:12:45.510 align:middle line:90% It's the most common. 00:12:45.510 --> 00:12:49.100 align:middle line:84% But there are a whole number of other so-called bear 00:12:49.100 --> 00:12:51.920 align:middle line:84% patterns that have been proposed which 00:12:51.920 --> 00:12:53.360 align:middle line:90% don't use this 2 by 2 tile. 00:12:53.360 --> 00:12:54.560 align:middle line:90% They have even bigger tiles. 00:12:54.560 --> 00:12:57.620 align:middle line:84% And the simplest one is red, green, and blue and clear. 00:12:57.620 --> 00:13:00.110 align:middle line:84% So, you get one pixel which gets all the wavelengths 00:13:00.110 --> 00:13:03.200 align:middle line:84% and then red, green, and blue interpolations. 00:13:03.200 --> 00:13:06.680 align:middle line:84% And I'm sure there are studies that 00:13:06.680 --> 00:13:09.440 align:middle line:84% compare various ones because if you don't get RGB, 00:13:09.440 --> 00:13:10.850 align:middle line:84% you need to do some of inversion. 00:13:10.850 --> 00:13:12.800 align:middle line:84% And you would get noise in that inversion. 00:13:12.800 --> 00:13:16.400 align:middle line:84% But if you use [INAUDIBLE] magenta, yellow, 00:13:16.400 --> 00:13:19.520 align:middle line:84% you would be able to get more light onto the sensor. 00:13:19.520 --> 00:13:21.920 align:middle line:90% So, I'm not very sure of-- 00:13:21.920 --> 00:13:23.840 align:middle line:84% I mean, I couldn't find any real studies 00:13:23.840 --> 00:13:26.313 align:middle line:84% that compare the various sensors or various-- 00:13:26.313 --> 00:13:27.980 align:middle line:84% PROFESSOR: Kodak has done a few studies. 00:13:27.980 --> 00:13:28.970 align:middle line:90% They have it on their website. 00:13:28.970 --> 00:13:29.845 align:middle line:90% ANKIT MOHAN: They do? 00:13:29.845 --> 00:13:32.508 align:middle line:90% I mean, it was always very-- 00:13:32.508 --> 00:13:33.050 align:middle line:90% I'm not sure. 00:13:33.050 --> 00:13:33.590 align:middle line:90% So, yeah. 00:13:33.590 --> 00:13:36.710 align:middle line:84% So, there are various trade offs between the different kinds 00:13:36.710 --> 00:13:38.360 align:middle line:84% of sensors but this is the only one 00:13:38.360 --> 00:13:39.890 align:middle line:84% that you find in practice right now. 00:13:39.890 --> 00:13:41.765 align:middle line:84% PROFESSOR: This will be a great class project 00:13:41.765 --> 00:13:45.408 align:middle line:84% by the way of figuring out which spectrum to choose depending 00:13:45.408 --> 00:13:45.950 align:middle line:90% on the scene. 00:13:45.950 --> 00:13:51.093 align:middle line:90% 00:13:51.093 --> 00:13:51.760 align:middle line:90% ANKIT MOHAN: OK. 00:13:51.760 --> 00:13:54.250 align:middle line:84% So, taking a step back from-- this 00:13:54.250 --> 00:13:57.160 align:middle line:84% was more a rehashing of what we already 00:13:57.160 --> 00:14:01.600 align:middle line:84% discussed before in terms of color sensing on cameras. 00:14:01.600 --> 00:14:03.760 align:middle line:84% It's important to look at what we are 00:14:03.760 --> 00:14:05.350 align:middle line:90% sensing when we sense light. 00:14:05.350 --> 00:14:08.320 align:middle line:84% And it's really a part of what's the electromagnetic spectrum 00:14:08.320 --> 00:14:11.620 align:middle line:84% that is actually much, much wider 00:14:11.620 --> 00:14:16.390 align:middle line:84% and has many other types of rays than what we are usually 00:14:16.390 --> 00:14:18.910 align:middle line:84% looking at in photography which is a visible spectrum. 00:14:18.910 --> 00:14:21.910 align:middle line:84% And the visible spectrum of light 00:14:21.910 --> 00:14:23.950 align:middle line:90% goes from 400 to 700 nanometers. 00:14:23.950 --> 00:14:25.600 align:middle line:84% Usually, when you talk about light, 00:14:25.600 --> 00:14:27.940 align:middle line:90% you talk in terms of wavelength. 00:14:27.940 --> 00:14:31.240 align:middle line:84% And in different-- in radio and microwave, 00:14:31.240 --> 00:14:33.610 align:middle line:84% you would probably talk in terms of frequency 00:14:33.610 --> 00:14:36.530 align:middle line:84% or in terahertz radiation and so on. 00:14:36.530 --> 00:14:40.750 align:middle line:84% But it's-- for visible light, you always say the wavelength 00:14:40.750 --> 00:14:44.813 align:middle line:84% is 400 to 700 nanometers going from blue-- from blue to red. 00:14:44.813 --> 00:14:46.480 align:middle line:84% And it's interesting that this is really 00:14:46.480 --> 00:14:50.410 align:middle line:84% the only wavelength region that is almost completely-- 00:14:50.410 --> 00:14:52.970 align:middle line:84% the atmosphere is almost completely transparent to it. 00:14:52.970 --> 00:14:55.930 align:middle line:84% So the sunlight that comes through is-- 00:14:55.930 --> 00:14:58.390 align:middle line:84% it's-- other than maybe radio waves, 00:14:58.390 --> 00:15:02.320 align:middle line:84% almost all other wavelengths are actually occluded 00:15:02.320 --> 00:15:03.590 align:middle line:90% by the atmosphere. 00:15:03.590 --> 00:15:08.260 align:middle line:84% And this is really the most of the natural illumination 00:15:08.260 --> 00:15:10.270 align:middle line:84% that you have is in this wavelength region 00:15:10.270 --> 00:15:12.850 align:middle line:84% which is probably why humans and most animals 00:15:12.850 --> 00:15:15.620 align:middle line:90% actually develop their-- 00:15:15.620 --> 00:15:18.970 align:middle line:84% are tuned for this region and not any other. 00:15:18.970 --> 00:15:20.890 align:middle line:84% And another sort of interesting thing 00:15:20.890 --> 00:15:23.960 align:middle line:84% is as you keep going away from this or shorter wavelength, 00:15:23.960 --> 00:15:26.110 align:middle line:84% the more dangerous out of the way it becomes. 00:15:26.110 --> 00:15:29.860 align:middle line:84% Already starting from UV, you start getting cancerous rays 00:15:29.860 --> 00:15:31.040 align:middle line:90% and so on. 00:15:31.040 --> 00:15:33.550 align:middle line:84% And then X-rays and gamma rays are even more so. 00:15:33.550 --> 00:15:35.350 align:middle line:84% But if you go for a larger wavelengths, 00:15:35.350 --> 00:15:37.330 align:middle line:90% they're usually harmless. 00:15:37.330 --> 00:15:40.120 align:middle line:84% PROFESSOR: And the way to remember that is wavelenth 00:15:40.120 --> 00:15:42.610 align:middle line:84% is inversely proportional to the frequency. 00:15:42.610 --> 00:15:45.460 align:middle line:84% And the product of the two, what's the number, and they 00:15:45.460 --> 00:15:47.895 align:middle line:84% multiply the wavelength of spectrum 00:15:47.895 --> 00:15:49.270 align:middle line:84% to the frequency of the spectrum. 00:15:49.270 --> 00:15:51.820 align:middle line:90% 00:15:51.820 --> 00:15:54.100 align:middle line:90% That's the speed of light. 00:15:54.100 --> 00:15:56.768 align:middle line:90% What's the speed of light? 00:15:56.768 --> 00:15:59.542 align:middle line:90% [INTERPOSING VOICES] 00:15:59.542 --> 00:16:00.750 align:middle line:90% PROFESSOR: But in this class? 00:16:00.750 --> 00:16:04.715 align:middle line:90% [INTERPOSING VOICES] 00:16:04.715 --> 00:16:08.670 align:middle line:84% And so the frequency is increasing 00:16:08.670 --> 00:16:12.000 align:middle line:84% as you go to the right, which means it has more energy, 00:16:12.000 --> 00:16:15.840 align:middle line:84% [? hnu, ?] which means it can penetrate deeper 00:16:15.840 --> 00:16:18.010 align:middle line:90% and damage more things. 00:16:18.010 --> 00:16:20.040 align:middle line:84% So, that's one easy way to remember 00:16:20.040 --> 00:16:21.420 align:middle line:90% what's going on the right. 00:16:21.420 --> 00:16:23.160 align:middle line:84% Unfortunately, the chart is flipped 00:16:23.160 --> 00:16:26.730 align:middle line:84% because we should think of limit increasing from left to right, 00:16:26.730 --> 00:16:29.590 align:middle line:84% but this is really showing the frequency increasing from left 00:16:29.590 --> 00:16:30.090 align:middle line:90% to right. 00:16:30.090 --> 00:16:32.760 align:middle line:90% 00:16:32.760 --> 00:16:35.270 align:middle line:84% ANKIT MOHAN: So yeah, I guess the point I want to make here 00:16:35.270 --> 00:16:36.685 align:middle line:90% is that we are-- 00:16:36.685 --> 00:16:38.060 align:middle line:84% it's only this region that we are 00:16:38.060 --> 00:16:40.280 align:middle line:84% looking at, which is really, really small compared 00:16:40.280 --> 00:16:43.740 align:middle line:84% to the whole spectrum, EM spectrum. 00:16:43.740 --> 00:16:46.730 align:middle line:84% and there is lots of interesting stuff going on, 00:16:46.730 --> 00:16:48.740 align:middle line:84% especially in thermal IR and thermal 00:16:48.740 --> 00:16:52.220 align:middle line:84% and even beyond that in using it for imaging. 00:16:52.220 --> 00:16:55.980 align:middle line:84% and we are using Wi-Fi for imaging also 00:16:55.980 --> 00:16:58.712 align:middle line:84% and things like that, which is support for gigahertz 00:16:58.712 --> 00:16:59.670 align:middle line:90% or something like that. 00:16:59.670 --> 00:17:03.290 align:middle line:84% So, you don't have to be constrained for even creating 00:17:03.290 --> 00:17:06.569 align:middle line:84% a photorealistic or a visual image of understanding what's 00:17:06.569 --> 00:17:07.069 align:middle line:90% around us. 00:17:07.069 --> 00:17:12.650 align:middle line:84% We shouldn't be limited to which will serve the visual spectrum. 00:17:12.650 --> 00:17:14.420 align:middle line:90% It's OK to think outside it. 00:17:14.420 --> 00:17:16.430 align:middle line:84% And I'll show a few examples of how 00:17:16.430 --> 00:17:17.990 align:middle line:84% thinking outside the visual spectrum 00:17:17.990 --> 00:17:21.380 align:middle line:84% actually enabled you to do a lot of things which you otherwise 00:17:21.380 --> 00:17:23.510 align:middle line:90% would not be able to do. 00:17:23.510 --> 00:17:27.050 align:middle line:84% So before again, I just wanted to talk 00:17:27.050 --> 00:17:28.520 align:middle line:90% about what a spectroscope is. 00:17:28.520 --> 00:17:29.970 align:middle line:90% And we have one of those here. 00:17:29.970 --> 00:17:33.890 align:middle line:84% So, I guess you could pass it around and so you-- 00:17:33.890 --> 00:17:37.350 align:middle line:90% 00:17:37.350 --> 00:17:42.810 align:middle line:84% so, what spectroscope really is nothing but a prism 00:17:42.810 --> 00:17:43.500 align:middle line:90% essentially. 00:17:43.500 --> 00:17:45.460 align:middle line:90% And a prism takes a signal. 00:17:45.460 --> 00:17:49.110 align:middle line:84% So, a prism is basically this optical element 00:17:49.110 --> 00:17:51.880 align:middle line:84% that bends any incoming ray of light. 00:17:51.880 --> 00:17:54.510 align:middle line:84% And but the interesting thing is because this is refraction, 00:17:54.510 --> 00:17:57.660 align:middle line:84% the refractive index of glass is actually slightly different 00:17:57.660 --> 00:17:58.290 align:middle line:90% based. 00:17:58.290 --> 00:18:00.670 align:middle line:84% It's a function of the wavelength of light. 00:18:00.670 --> 00:18:04.950 align:middle line:84% So, if you build a prism of the right material, right kind 00:18:04.950 --> 00:18:08.730 align:middle line:84% of glass, you can have it have a huge disparity 00:18:08.730 --> 00:18:13.300 align:middle line:84% in the refractive index between the 400 and 700 nanometers. 00:18:13.300 --> 00:18:17.790 align:middle line:84% And so when the light bend coming out of a prism. 00:18:17.790 --> 00:18:19.860 align:middle line:84% So, it says grating here but can be a prism 00:18:19.860 --> 00:18:21.630 align:middle line:90% or creating any of those. 00:18:21.630 --> 00:18:26.080 align:middle line:84% When the light went from here, the red and blue-- 00:18:26.080 --> 00:18:28.080 align:middle line:84% red, blue, and green, the different wavelengths, 00:18:28.080 --> 00:18:29.860 align:middle line:84% actually bend in different directions. 00:18:29.860 --> 00:18:32.160 align:middle line:84% And if you have a detector placed in front of it, 00:18:32.160 --> 00:18:36.240 align:middle line:84% you can sense the intensity of the blue ray, of the green ray, 00:18:36.240 --> 00:18:37.660 align:middle line:90% and of the red ray separately. 00:18:37.660 --> 00:18:41.130 align:middle line:84% And you can sort of decompose an incoming source of light 00:18:41.130 --> 00:18:42.600 align:middle line:84% into its constituent wavelengths. 00:18:42.600 --> 00:18:44.070 align:middle line:90% PROFESSOR: You have a grating? 00:18:44.070 --> 00:18:44.882 align:middle line:90% ANKIT MOHAN: Yeah. 00:18:44.882 --> 00:18:47.340 align:middle line:84% PROFESSOR: So, I'll pass this around and all you have to do 00:18:47.340 --> 00:18:49.800 align:middle line:90% is look through this hole here. 00:18:49.800 --> 00:18:54.360 align:middle line:84% And all it has is a slit in front. 00:18:54.360 --> 00:18:57.060 align:middle line:84% so the best path forward was to look at one 00:18:57.060 --> 00:18:59.370 align:middle line:90% of the bright lights up here. 00:18:59.370 --> 00:19:02.730 align:middle line:84% And very conveniently, there is a scale 00:19:02.730 --> 00:19:06.910 align:middle line:84% on this side that goes from 400 nanometers to 700 nanometers. 00:19:06.910 --> 00:19:09.730 align:middle line:84% So, the idea is that you can look at any point in the world 00:19:09.730 --> 00:19:12.340 align:middle line:84% and you can see its spectrum right away. 00:19:12.340 --> 00:19:16.590 align:middle line:84% So, unfortunately, we don't have fluorescent lights. 00:19:16.590 --> 00:19:18.726 align:middle line:90% ANKIT MOHAN: Actually, we do. 00:19:18.726 --> 00:19:23.972 align:middle line:90% STUDENT: [INAUDIBLE] 00:19:23.972 --> 00:19:24.680 align:middle line:90% PROFESSOR: Right. 00:19:24.680 --> 00:19:27.200 align:middle line:84% And you'll realize the [INAUDIBLE] is actually 00:19:27.200 --> 00:19:27.800 align:middle line:90% very spiky. 00:19:27.800 --> 00:19:31.540 align:middle line:84% It has like sharp blue and then a little bit of green and very, 00:19:31.540 --> 00:19:32.978 align:middle line:90% very annoying spectrum. 00:19:32.978 --> 00:19:33.770 align:middle line:90% It's not very nice. 00:19:33.770 --> 00:19:36.200 align:middle line:90% It's more like the sunlight. 00:19:36.200 --> 00:19:38.360 align:middle line:90% And just pass it around. 00:19:38.360 --> 00:19:40.100 align:middle line:84% I was going to pass around the flashlight 00:19:40.100 --> 00:19:43.610 align:middle line:84% but I'm not sure how many people have 00:19:43.610 --> 00:19:45.020 align:middle line:90% good hand-eye coordination. 00:19:45.020 --> 00:19:47.318 align:middle line:90% 00:19:47.318 --> 00:19:48.860 align:middle line:84% ANKIT MOHAN: There's also this, which 00:19:48.860 --> 00:19:51.950 align:middle line:90% is diffraction grating and it-- 00:19:51.950 --> 00:19:53.960 align:middle line:84% for the purpose of this dispersion, 00:19:53.960 --> 00:19:55.220 align:middle line:90% it's very similar to a prism. 00:19:55.220 --> 00:19:56.595 align:middle line:84% So, if you look through this, you 00:19:56.595 --> 00:19:59.240 align:middle line:84% will be able to see a whole rainbow of colors 00:19:59.240 --> 00:20:02.510 align:middle line:90% around any bright light. 00:20:02.510 --> 00:20:05.210 align:middle line:84% PROFESSOR: So, when you're looking at this, 00:20:05.210 --> 00:20:09.050 align:middle line:84% look at it through a particular angle, a particular orientation 00:20:09.050 --> 00:20:10.640 align:middle line:90% and then rotate it. 00:20:10.640 --> 00:20:15.148 align:middle line:84% And as you rotate, you'll realize that it will fall down. 00:20:15.148 --> 00:20:16.940 align:middle line:84% STUDENT: Speaking of hand-eye coordination. 00:20:16.940 --> 00:20:17.882 align:middle line:90% PROFESSOR: Exactly. 00:20:17.882 --> 00:20:18.590 align:middle line:90% I have an excuse. 00:20:18.590 --> 00:20:19.640 align:middle line:90% I was sick this week. 00:20:19.640 --> 00:20:22.370 align:middle line:90% 00:20:22.370 --> 00:20:24.420 align:middle line:84% You'll see that the image actually shifts around. 00:20:24.420 --> 00:20:26.420 align:middle line:84% So, if I'm looking at Daniel, I see him. 00:20:26.420 --> 00:20:31.160 align:middle line:84% Then I see a red copy of him and a blue copy of him shifted. 00:20:31.160 --> 00:20:34.260 align:middle line:84% I knew there was more than one of you. 00:20:34.260 --> 00:20:36.080 align:middle line:90% And if I rotate, it rotates. 00:20:36.080 --> 00:20:39.530 align:middle line:84% And just remember this principle when we come back and talk, 00:20:39.530 --> 00:20:42.050 align:middle line:84% when Michael starts talking about different mechanisms 00:20:42.050 --> 00:20:43.640 align:middle line:90% for exploiting wavelength. 00:20:43.640 --> 00:20:45.280 align:middle line:84% Again, a great class project idea. 00:20:45.280 --> 00:20:49.790 align:middle line:90% 00:20:49.790 --> 00:20:51.290 align:middle line:90% ANKIT MOHAN: Yeah. 00:20:51.290 --> 00:20:55.940 align:middle line:84% So, this here just shows one of our typical spectrogram. 00:20:55.940 --> 00:20:58.770 align:middle line:84% Kind of looks like it's-- you have the wavelength over here 00:20:58.770 --> 00:21:01.470 align:middle line:84% and the intensity over here on the y-axis. 00:21:01.470 --> 00:21:06.560 align:middle line:84% So, you can see those spikes over here at 550 nanometer 00:21:06.560 --> 00:21:08.300 align:middle line:90% and then again, 600 something. 00:21:08.300 --> 00:21:10.610 align:middle line:84% So this is probably a fluorescent light 00:21:10.610 --> 00:21:11.868 align:middle line:90% is kind of similar I guess. 00:21:11.868 --> 00:21:13.410 align:middle line:84% PROFESSOR: Maybe we can pass around-- 00:21:13.410 --> 00:21:17.120 align:middle line:90% [INTERPOSING VOICES] 00:21:17.120 --> 00:21:18.017 align:middle line:90% LED light. 00:21:18.017 --> 00:21:18.850 align:middle line:90% [INTERPOSING VOICES] 00:21:18.850 --> 00:21:19.955 align:middle line:90% ANKIT MOHAN: Will be very spiky. 00:21:19.955 --> 00:21:20.622 align:middle line:90% PROFESSOR: Yeah. 00:21:20.622 --> 00:21:23.930 align:middle line:90% That'll be really good to see. 00:21:23.930 --> 00:21:26.663 align:middle line:84% Take a look at this, [INAUDIBLE].. 00:21:26.663 --> 00:21:27.330 align:middle line:90% ANKIT MOHAN: OK. 00:21:27.330 --> 00:21:30.870 align:middle line:84% So, while one of you is looking-- so, 00:21:30.870 --> 00:21:33.150 align:middle line:84% this is stuff that Michael is going to get 00:21:33.150 --> 00:21:34.660 align:middle line:90% into much more detail later. 00:21:34.660 --> 00:21:36.210 align:middle line:90% But I just wanted to mention-- 00:21:36.210 --> 00:21:39.165 align:middle line:84% are you going to talk about [INAUDIBLE]?? 00:21:39.165 --> 00:21:41.423 align:middle line:84% MICHAEL: It won't hurt the do it twice so go for it. 00:21:41.423 --> 00:21:42.090 align:middle line:90% ANKIT MOHAN: OK. 00:21:42.090 --> 00:21:44.340 align:middle line:84% So, I mean, it's just something which 00:21:44.340 --> 00:21:48.090 align:middle line:84% is very high level of what he's actually going to talk about 00:21:48.090 --> 00:21:50.850 align:middle line:84% is the spectroscope is really imaging 00:21:50.850 --> 00:21:54.900 align:middle line:84% the multiple spectrums of a single point in space 00:21:54.900 --> 00:21:56.970 align:middle line:90% like a light source over here. 00:21:56.970 --> 00:21:59.460 align:middle line:84% If you have a full scene, you are not 00:21:59.460 --> 00:22:01.590 align:middle line:84% going to get much out of a spectroscope. 00:22:01.590 --> 00:22:03.420 align:middle line:84% And what you need to do then is what's 00:22:03.420 --> 00:22:06.180 align:middle line:84% called multispectral imaging or hyperspectral imaging 00:22:06.180 --> 00:22:08.430 align:middle line:84% depending on how many spectrums you end up getting. 00:22:08.430 --> 00:22:12.940 align:middle line:84% And so there are these standard ways of doing that. 00:22:12.940 --> 00:22:16.530 align:middle line:84% And I just wanted to briefly mention what those are. 00:22:16.530 --> 00:22:19.320 align:middle line:84% So, usually this kind of multispectral imaging 00:22:19.320 --> 00:22:22.407 align:middle line:84% is very popular and remote sensing kind of applications 00:22:22.407 --> 00:22:24.240 align:middle line:84% where you have a plane or a satellite that's 00:22:24.240 --> 00:22:25.698 align:middle line:84% flying over the region and you want 00:22:25.698 --> 00:22:29.250 align:middle line:84% to get the multispectral or hyperspectral image or data 00:22:29.250 --> 00:22:32.230 align:middle line:84% set of what's on the ground underneath. 00:22:32.230 --> 00:22:34.530 align:middle line:84% So, what we have here is a lens kind 00:22:34.530 --> 00:22:37.680 align:middle line:84% of thing, which is on the plane image sensor behind it 00:22:37.680 --> 00:22:40.600 align:middle line:84% and the object space, which is on the ground. 00:22:40.600 --> 00:22:43.890 align:middle line:84% And so the most straightforward way of doing this 00:22:43.890 --> 00:22:45.555 align:middle line:90% is what's called-- 00:22:45.555 --> 00:22:48.120 align:middle line:90% 00:22:48.120 --> 00:22:50.430 align:middle line:84% so, even this simple imaging where 00:22:50.430 --> 00:22:52.380 align:middle line:84% you're not imaging the complete spectrum, 00:22:52.380 --> 00:22:54.220 align:middle line:90% you can do it in various ways. 00:22:54.220 --> 00:22:57.300 align:middle line:84% The first way is where you have a lens which 00:22:57.300 --> 00:23:01.050 align:middle line:84% completely images the object space onto an image or 2D 00:23:01.050 --> 00:23:01.990 align:middle line:90% sensor. 00:23:01.990 --> 00:23:05.220 align:middle line:84% So, this is what just a traditional camera looks like. 00:23:05.220 --> 00:23:07.740 align:middle line:84% You capture the whole scene in one go. 00:23:07.740 --> 00:23:10.380 align:middle line:84% Another way of doing this is what's called a push broom. 00:23:10.380 --> 00:23:12.300 align:middle line:84% And a push broom, the way I think 00:23:12.300 --> 00:23:15.120 align:middle line:84% of it it's you're pushing forward 00:23:15.120 --> 00:23:19.450 align:middle line:84% as the platform or the plane is moving in that direction. 00:23:19.450 --> 00:23:22.810 align:middle line:84% So first, when the plane is at a particular location, 00:23:22.810 --> 00:23:26.670 align:middle line:84% you are imaging this line on the scene. 00:23:26.670 --> 00:23:29.220 align:middle line:84% And then in the next instance, you will image this line 00:23:29.220 --> 00:23:31.420 align:middle line:84% and then you will image this line and so on. 00:23:31.420 --> 00:23:34.080 align:middle line:84% So as you're moving forward, you are imaging one line 00:23:34.080 --> 00:23:36.960 align:middle line:90% that's [INAUDIBLE] you. 00:23:36.960 --> 00:23:41.172 align:middle line:84% The other one is called the whisk broom in which you are-- 00:23:41.172 --> 00:23:42.630 align:middle line:84% while you're moving forward, you're 00:23:42.630 --> 00:23:44.970 align:middle line:84% going from left to right and from right to left. 00:23:44.970 --> 00:23:47.170 align:middle line:84% And you're doing this whisk broom kind of thing, 00:23:47.170 --> 00:23:48.720 align:middle line:90% as opposed to a push room. 00:23:48.720 --> 00:23:51.690 align:middle line:84% And what you do with this whisk broom 00:23:51.690 --> 00:23:54.600 align:middle line:84% is that when you are imaging this one element, you-- 00:23:54.600 --> 00:23:56.910 align:middle line:84% instead of just imaging onto one sensor, 00:23:56.910 --> 00:23:58.650 align:middle line:84% you pass the light through a prism, 00:23:58.650 --> 00:24:02.160 align:middle line:84% and you get a whole spectrum similar to a spectroscope. 00:24:02.160 --> 00:24:06.990 align:middle line:84% And this gives you the complete spectral characteristics 00:24:06.990 --> 00:24:10.313 align:middle line:84% of this one element or one point on scene. 00:24:10.313 --> 00:24:11.730 align:middle line:84% And then in the next instance, you 00:24:11.730 --> 00:24:14.520 align:middle line:84% are going to sense the next element right next to it 00:24:14.520 --> 00:24:19.480 align:middle line:84% as your whisk broom is going from left to right and so on. 00:24:19.480 --> 00:24:23.550 align:middle line:84% So, you're going to sense the whole scene. 00:24:23.550 --> 00:24:25.830 align:middle line:84% The way the push broom hyperspectral sensor 00:24:25.830 --> 00:24:27.330 align:middle line:90% works is that you-- 00:24:27.330 --> 00:24:29.850 align:middle line:84% so in this case, you need just 1D sensor 00:24:29.850 --> 00:24:33.990 align:middle line:84% in order to capture the complete data 00:24:33.990 --> 00:24:36.450 align:middle line:84% set of the hyperspectral image of the scene. 00:24:36.450 --> 00:24:39.630 align:middle line:84% Another way of doing it is using the push broom where 00:24:39.630 --> 00:24:41.490 align:middle line:90% instead of cap-- 00:24:41.490 --> 00:24:43.758 align:middle line:84% so, you put a prism in front of each 00:24:43.758 --> 00:24:46.050 align:middle line:84% of these elements, the image of each of these elements, 00:24:46.050 --> 00:24:47.610 align:middle line:90% and then you have a 2D sensor. 00:24:47.610 --> 00:24:49.980 align:middle line:84% And the 2D sensor would sense along 00:24:49.980 --> 00:24:53.850 align:middle line:84% the x-axis is the point in space or the point in scene. 00:24:53.850 --> 00:24:57.090 align:middle line:84% And along the y-axis is the various spectral bands 00:24:57.090 --> 00:24:59.460 align:middle line:84% that you have for your hyperspectral imager. 00:24:59.460 --> 00:25:01.470 align:middle line:84% And then again, in the next instance, 00:25:01.470 --> 00:25:06.210 align:middle line:84% you will sense the next [INAUDIBLE] row in the scene. 00:25:06.210 --> 00:25:08.370 align:middle line:84% A third way of doing it is basically 00:25:08.370 --> 00:25:11.130 align:middle line:84% something that's similar to the traditional camera 00:25:11.130 --> 00:25:13.140 align:middle line:84% is to put a filter in front of your lens 00:25:13.140 --> 00:25:15.810 align:middle line:84% and change the filter that's in front of your lens. 00:25:15.810 --> 00:25:17.880 align:middle line:84% Either have that color wheel or have 00:25:17.880 --> 00:25:20.650 align:middle line:84% a tunable filter which is-- whose response is changing. 00:25:20.650 --> 00:25:23.010 align:middle line:84% But at any given instant, you have-- you're either 00:25:23.010 --> 00:25:25.202 align:middle line:90% sensing green red or blue. 00:25:25.202 --> 00:25:27.660 align:middle line:84% And then in the next instance, you're sensing the next one. 00:25:27.660 --> 00:25:29.400 align:middle line:84% It's similar to the first image we 00:25:29.400 --> 00:25:32.250 align:middle line:84% saw when I started the talk of capturing 00:25:32.250 --> 00:25:35.160 align:middle line:84% multiple images with different wavelength filters in front 00:25:35.160 --> 00:25:36.210 align:middle line:90% of the camera. 00:25:36.210 --> 00:25:37.450 align:middle line:90% It's similar to that. 00:25:37.450 --> 00:25:40.800 align:middle line:84% So, you capture the whole scene in one instant 00:25:40.800 --> 00:25:44.290 align:middle line:90% but only for one wavelength. 00:25:44.290 --> 00:25:47.040 align:middle line:84% So this summarizes what I was talking about 00:25:47.040 --> 00:25:49.920 align:middle line:84% that you have this what's called the data cube or the object 00:25:49.920 --> 00:25:54.495 align:middle line:84% cube, which is the x and y are the same coordinates, x y. 00:25:54.495 --> 00:25:59.130 align:middle line:84% And lambda is the wavelength where it goes from 400 to 700 00:25:59.130 --> 00:26:02.280 align:middle line:84% or slightly beyond those extremes. 00:26:02.280 --> 00:26:04.840 align:middle line:84% You might also have near IR and so on. 00:26:04.840 --> 00:26:06.810 align:middle line:84% But this is what you want to get. 00:26:06.810 --> 00:26:10.150 align:middle line:84% And unfortunately-- so this is a 3D object. 00:26:10.150 --> 00:26:12.840 align:middle line:84% And what you-- unfortunately, our sensor 00:26:12.840 --> 00:26:16.030 align:middle line:84% can be at most two dimensional surface. 00:26:16.030 --> 00:26:19.650 align:middle line:84% So, you want to somehow get the three dimensional data set 00:26:19.650 --> 00:26:21.960 align:middle line:90% onto a two dimensional sensor. 00:26:21.960 --> 00:26:25.090 align:middle line:84% And there are various ways in which you can do that. 00:26:25.090 --> 00:26:28.260 align:middle line:84% And you can-- the most traditional, most obvious way 00:26:28.260 --> 00:26:30.690 align:middle line:84% of doing it is to have a third dimension be time. 00:26:30.690 --> 00:26:32.580 align:middle line:84% So, at any one given instant, you 00:26:32.580 --> 00:26:35.310 align:middle line:84% are either getting a slice like this or a slice 00:26:35.310 --> 00:26:39.060 align:middle line:84% like this or a slice like this either along x or along y 00:26:39.060 --> 00:26:41.430 align:middle line:90% or along the wavelength. 00:26:41.430 --> 00:26:43.410 align:middle line:84% And that in the next instant, you 00:26:43.410 --> 00:26:45.315 align:middle line:90% get the next slice and so on. 00:26:45.315 --> 00:26:47.440 align:middle line:84% And then you combine all this information together. 00:26:47.440 --> 00:26:51.240 align:middle line:84% So over a period of time, you build a complete object cube 00:26:51.240 --> 00:26:52.530 align:middle line:90% or data set. 00:26:52.530 --> 00:26:54.930 align:middle line:84% So, this one is where you're using a filter. 00:26:54.930 --> 00:26:58.110 align:middle line:84% And this is where you're using a whisk broom or a push broom 00:26:58.110 --> 00:27:00.810 align:middle line:84% depending on how you name them, x and y. 00:27:00.810 --> 00:27:03.450 align:middle line:84% So, this is sort of the more traditional way of doing 00:27:03.450 --> 00:27:04.530 align:middle line:90% multi-spectral scanning. 00:27:04.530 --> 00:27:07.560 align:middle line:84% And Michael is going to talk about even more 00:27:07.560 --> 00:27:09.300 align:middle line:84% interesting fancier ways of doing it 00:27:09.300 --> 00:27:10.680 align:middle line:90% by just taking projections. 00:27:10.680 --> 00:27:11.970 align:middle line:90% But I let him get into that. 00:27:11.970 --> 00:27:15.260 align:middle line:90% 00:27:15.260 --> 00:27:18.760 align:middle line:84% So, I think Ramesh briefly mentioned thermal imaging. 00:27:18.760 --> 00:27:24.200 align:middle line:84% And can we get the lights off or? 00:27:24.200 --> 00:27:25.750 align:middle line:84% So, this is what-- for those of you 00:27:25.750 --> 00:27:28.980 align:middle line:84% who've never seen a thermal camera, we have one upstairs. 00:27:28.980 --> 00:27:30.730 align:middle line:84% I think [INAUDIBLE] welcome and play with. 00:27:30.730 --> 00:27:34.120 align:middle line:84% But this is what images through our typical thermal imager 00:27:34.120 --> 00:27:36.040 align:middle line:90% or thermal camera look like. 00:27:36.040 --> 00:27:38.080 align:middle line:84% This is sensing and usually in the wavelengths 00:27:38.080 --> 00:27:40.840 align:middle line:84% in the range of 1 micron to about 10 microns 00:27:40.840 --> 00:27:42.220 align:middle line:90% or 15 microns in that range. 00:27:42.220 --> 00:27:45.820 align:middle line:84% That's the range in which humans show up as warm bodies usually. 00:27:45.820 --> 00:27:48.140 align:middle line:90% So, 6 to 8 micron kind of range. 00:27:48.140 --> 00:27:51.400 align:middle line:90% But you can also have-- 00:27:51.400 --> 00:27:53.710 align:middle line:84% actually, hot bodies, warm bodies 00:27:53.710 --> 00:27:58.390 align:middle line:84% show up like heat seeking missiles and so on. 00:27:58.390 --> 00:28:01.090 align:middle line:84% This is something that's explored 00:28:01.090 --> 00:28:03.280 align:middle line:90% a lot in the defense industry. 00:28:03.280 --> 00:28:07.180 align:middle line:84% This use of high resolution very fast thermal cameras, 00:28:07.180 --> 00:28:10.180 align:middle line:84% but it's something that's very rapidly coming 00:28:10.180 --> 00:28:12.670 align:middle line:84% and into the other applications also. 00:28:12.670 --> 00:28:16.330 align:middle line:84% There was recently an article in Time. 00:28:16.330 --> 00:28:19.930 align:middle line:84% I think that they were looking at the various-- 00:28:19.930 --> 00:28:23.500 align:middle line:84% the thermal profile of a house from outside 00:28:23.500 --> 00:28:26.560 align:middle line:84% to find out where it's leaking and where the heat is escaping. 00:28:26.560 --> 00:28:29.020 align:middle line:84% And then they're using these kind of thermal cameras 00:28:29.020 --> 00:28:31.750 align:middle line:84% to actually test for whether pipes are leaking 00:28:31.750 --> 00:28:34.535 align:middle line:84% and your HVAC is working properly or not 00:28:34.535 --> 00:28:35.410 align:middle line:90% and things like that. 00:28:35.410 --> 00:28:37.600 align:middle line:84% So, it's finding lots of applications 00:28:37.600 --> 00:28:39.790 align:middle line:84% in areas other than just traditional defense. 00:28:39.790 --> 00:28:40.430 align:middle line:90% And so on. 00:28:40.430 --> 00:28:42.460 align:middle line:84% And that's one of the reasons why 00:28:42.460 --> 00:28:44.500 align:middle line:84% slowly the price of these cameras which 00:28:44.500 --> 00:28:50.180 align:middle line:84% used to be about $20,000 each is hopefully coming down. 00:28:50.180 --> 00:28:54.430 align:middle line:84% This is another image of just thermal images which 00:28:54.430 --> 00:28:55.665 align:middle line:90% I thought was interesting. 00:28:55.665 --> 00:28:57.040 align:middle line:84% The first one is-- what's showing 00:28:57.040 --> 00:28:59.860 align:middle line:84% is that when you think of thermal light, 00:28:59.860 --> 00:29:02.500 align:middle line:84% it's actually quite different from when you're 00:29:02.500 --> 00:29:04.000 align:middle line:90% looking at visible light. 00:29:04.000 --> 00:29:07.930 align:middle line:84% And one example, I think Ramesh already mentioned 00:29:07.930 --> 00:29:11.650 align:middle line:84% is that of glass, that glass appears opaque in thermal IR. 00:29:11.650 --> 00:29:13.600 align:middle line:84% So, you can't look through glass. 00:29:13.600 --> 00:29:16.990 align:middle line:84% And there are other objects which may appear completely-- 00:29:16.990 --> 00:29:18.970 align:middle line:84% excuse me, transparent in thermal IR 00:29:18.970 --> 00:29:20.770 align:middle line:90% but are opaque in visible light. 00:29:20.770 --> 00:29:22.910 align:middle line:84% This is one example, which is sort of interesting, 00:29:22.910 --> 00:29:25.240 align:middle line:84% you have this fridge, which is brushed metal. 00:29:25.240 --> 00:29:28.780 align:middle line:84% And so in reflection, you see this very diffuse reflection. 00:29:28.780 --> 00:29:31.370 align:middle line:84% You can't really see what's on the other side. 00:29:31.370 --> 00:29:33.850 align:middle line:84% But if you just use thermal IR, you can very clearly 00:29:33.850 --> 00:29:37.570 align:middle line:84% see there is this rice oven on the other side that's 00:29:37.570 --> 00:29:38.590 align:middle line:90% really, really hot. 00:29:38.590 --> 00:29:40.210 align:middle line:84% And the reflection is nice and sharp 00:29:40.210 --> 00:29:42.250 align:middle line:84% because the wavelength that you're using 00:29:42.250 --> 00:29:45.940 align:middle line:84% is now not 700 nanometers but it's something much larger. 00:29:45.940 --> 00:29:48.580 align:middle line:84% And the surface is actually very smooth 00:29:48.580 --> 00:29:50.830 align:middle line:84% when you look at that wavelength. 00:29:50.830 --> 00:29:53.350 align:middle line:84% But when you're looking at it in the visible spectrum, 00:29:53.350 --> 00:29:55.930 align:middle line:84% it appears very diffuse and you can hardly 00:29:55.930 --> 00:29:57.225 align:middle line:90% see what the reflection is. 00:29:57.225 --> 00:29:58.600 align:middle line:84% PROFESSOR: Behaves Like a mirror. 00:29:58.600 --> 00:30:01.990 align:middle line:84% ANKIT MOHAN: It behaves like a mirror in the thermal IR 00:30:01.990 --> 00:30:04.823 align:middle line:84% range because of the difference in wavelength. 00:30:04.823 --> 00:30:06.490 align:middle line:84% So, this is sort of an interesting thing 00:30:06.490 --> 00:30:10.060 align:middle line:84% to think about of applications where you can-- imagine 00:30:10.060 --> 00:30:12.700 align:middle line:84% you had one of these thermal imaging cameras on each cell 00:30:12.700 --> 00:30:14.620 align:middle line:84% phone right next to your normal camera. 00:30:14.620 --> 00:30:15.910 align:middle line:90% What could you do with it? 00:30:15.910 --> 00:30:18.580 align:middle line:84% Or let's say you had it right next to your webcam 00:30:18.580 --> 00:30:21.880 align:middle line:90% on your laptop, can you do use-- 00:30:21.880 --> 00:30:24.260 align:middle line:84% do something interesting with it? 00:30:24.260 --> 00:30:28.340 align:middle line:90% Another one is this paper from-- 00:30:28.340 --> 00:30:29.578 align:middle line:90% Colorado-- no, University-- 00:30:29.578 --> 00:30:30.370 align:middle line:90% PROFESSOR: Houston. 00:30:30.370 --> 00:30:31.537 align:middle line:90% ANKIT MOHAN: Houston, right. 00:30:31.537 --> 00:30:35.523 align:middle line:84% Where they use thermal cameras to detect for lie detection. 00:30:35.523 --> 00:30:36.940 align:middle line:84% And they have this paper in Nature 00:30:36.940 --> 00:30:41.590 align:middle line:84% where they analyze the region around the eyes 00:30:41.590 --> 00:30:43.090 align:middle line:84% and how that changes when someone is 00:30:43.090 --> 00:30:44.950 align:middle line:90% lying in the thermal IR range. 00:30:44.950 --> 00:30:47.740 align:middle line:84% And they claim that they're getting as good performance 00:30:47.740 --> 00:30:51.437 align:middle line:90% as a traditional lie detector. 00:30:51.437 --> 00:30:53.020 align:middle line:84% PROFESSOR: It's not easy to reproduce, 00:30:53.020 --> 00:30:55.000 align:middle line:90% unfortunately, the results. 00:30:55.000 --> 00:30:58.120 align:middle line:90% But it looks pretty interesting. 00:30:58.120 --> 00:31:03.730 align:middle line:84% And the whole concept is that the blood veins pump more blood 00:31:03.730 --> 00:31:05.390 align:middle line:90% as your emotions change. 00:31:05.390 --> 00:31:08.740 align:middle line:84% So, as long as you can detect subcutaneous changes 00:31:08.740 --> 00:31:15.437 align:middle line:84% in blood flow, you can detect the correlated emotions. 00:31:15.437 --> 00:31:17.770 align:middle line:84% ANKIT MOHAN: I think the difference is really very, very 00:31:17.770 --> 00:31:20.260 align:middle line:84% subtle, and it's greatly magnified in this image 00:31:20.260 --> 00:31:20.960 align:middle line:90% that they show. 00:31:20.960 --> 00:31:23.380 align:middle line:84% So, when we tried to do this, we couldn't spot 00:31:23.380 --> 00:31:24.640 align:middle line:90% any difference [INAUDIBLE]. 00:31:24.640 --> 00:31:26.890 align:middle line:84% PROFESSOR: Of course, nobody lies in our group either. 00:31:26.890 --> 00:31:29.240 align:middle line:90% 00:31:29.240 --> 00:31:31.210 align:middle line:84% ANKIT MOHAN: So, one other thing-- 00:31:31.210 --> 00:31:33.040 align:middle line:84% when we were doing this last year, 00:31:33.040 --> 00:31:35.380 align:middle line:84% someone talked about near-infrared photography. 00:31:35.380 --> 00:31:37.422 align:middle line:84% And this is more from a photography point of view 00:31:37.422 --> 00:31:41.110 align:middle line:84% that actually, it turns out most of the CCD sensors are-- 00:31:41.110 --> 00:31:44.170 align:middle line:84% they're actually sensitive to neon IR or infrared 00:31:44.170 --> 00:31:49.150 align:middle line:84% that's just after 700 nanometer, from 700 to about 1 micron. 00:31:49.150 --> 00:31:51.190 align:middle line:90% And they are sensitive to that. 00:31:51.190 --> 00:31:54.670 align:middle line:84% But in fact, most manufacturers put an IR block filter 00:31:54.670 --> 00:31:57.610 align:middle line:84% on top of the sensor that blocks anything that's 00:31:57.610 --> 00:31:59.360 align:middle line:90% greater than 700 nanometers. 00:31:59.360 --> 00:32:02.020 align:middle line:84% So, what you can do is remove that IR block 00:32:02.020 --> 00:32:04.850 align:middle line:84% and you can then capture nice 3D images like this. 00:32:04.850 --> 00:32:08.410 align:middle line:84% This of course, is all-- they're all fake images because you 00:32:08.410 --> 00:32:10.420 align:middle line:84% don't really get any sense of color 00:32:10.420 --> 00:32:12.980 align:middle line:84% once you go beyond 700 nanometers. 00:32:12.980 --> 00:32:16.150 align:middle line:84% So people usually just fill in fake colors 00:32:16.150 --> 00:32:20.870 align:middle line:84% based on an original visual image or something like that. 00:32:20.870 --> 00:32:23.710 align:middle line:84% But another place where you can get these colors 00:32:23.710 --> 00:32:26.320 align:middle line:90% is if you use IR film. 00:32:26.320 --> 00:32:28.180 align:middle line:84% Kodak has this color IR film which 00:32:28.180 --> 00:32:30.520 align:middle line:84% reacts differently to different wavelengths 00:32:30.520 --> 00:32:32.360 align:middle line:90% beyond the 700 nanometers. 00:32:32.360 --> 00:32:34.805 align:middle line:84% And that gives you these interesting colors. 00:32:34.805 --> 00:32:36.680 align:middle line:84% PROFESSOR: So these are not nighttime photos. 00:32:36.680 --> 00:32:40.000 align:middle line:84% These are just daytime photos with some pseudo color 00:32:40.000 --> 00:32:41.850 align:middle line:90% superimposed. 00:32:41.850 --> 00:32:43.600 align:middle line:84% ANKIT MOHAN: So, the one interesting thing 00:32:43.600 --> 00:32:46.400 align:middle line:84% is that I think sky becomes really, 00:32:46.400 --> 00:32:47.470 align:middle line:90% really black and opaque. 00:32:47.470 --> 00:32:50.650 align:middle line:84% So, that's because if you remember, 00:32:50.650 --> 00:32:53.900 align:middle line:84% sky does not allow IR to come through as much. 00:32:53.900 --> 00:32:57.500 align:middle line:84% But also, any vegetation becomes very bright and white. 00:32:57.500 --> 00:33:01.630 align:middle line:84% And that's why you have the snow like effect on trees and things 00:33:01.630 --> 00:33:02.300 align:middle line:90% like that. 00:33:02.300 --> 00:33:07.150 align:middle line:84% But your barks of the tree and the food actually does not-- 00:33:07.150 --> 00:33:08.620 align:middle line:90% it actually absorbs more light. 00:33:08.620 --> 00:33:10.420 align:middle line:84% It does not reflect so much back, 00:33:10.420 --> 00:33:13.655 align:middle line:90% so it's just interesting things. 00:33:13.655 --> 00:33:16.340 align:middle line:90% 00:33:16.340 --> 00:33:18.610 align:middle line:84% And I mentioned this briefly earlier, 00:33:18.610 --> 00:33:21.280 align:middle line:84% one of the biggest applications of this thing 00:33:21.280 --> 00:33:23.900 align:middle line:84% has traditionally been in remote sensing also. 00:33:23.900 --> 00:33:27.820 align:middle line:84% And by capturing multi-spectral or hyperspectral images 00:33:27.820 --> 00:33:30.940 align:middle line:84% of scene of especially vegetation, 00:33:30.940 --> 00:33:33.940 align:middle line:84% you can actually classify what crop 00:33:33.940 --> 00:33:37.240 align:middle line:84% is what and what kind of-- what's the role 00:33:37.240 --> 00:33:39.760 align:middle line:84% and where you have plantations and what kind 00:33:39.760 --> 00:33:42.640 align:middle line:84% of plantations over here they're able to distinguish between all 00:33:42.640 --> 00:33:44.300 align:middle line:90% these different kinds of crops. 00:33:44.300 --> 00:33:46.390 align:middle line:84% And this is something you can do if you 00:33:46.390 --> 00:33:49.200 align:middle line:84% have enough resolution in the multiple spectrums that you're 00:33:49.200 --> 00:33:52.920 align:middle line:84% getting just because each vegetation or tree actually 00:33:52.920 --> 00:33:57.430 align:middle line:84% has a very different reflectance profile when you look at it. 00:33:57.430 --> 00:34:00.630 align:middle line:84% In other words, they both appear green to the human eye. 00:34:00.630 --> 00:34:02.610 align:middle line:84% If you look at the actual spectrum response, 00:34:02.610 --> 00:34:03.640 align:middle line:90% it's quite different. 00:34:03.640 --> 00:34:07.740 align:middle line:84% Then you can distinguish between different materials 00:34:07.740 --> 00:34:10.302 align:middle line:90% based on that. 00:34:10.302 --> 00:34:11.969 align:middle line:84% STUDENT: This is not based on IR though? 00:34:11.969 --> 00:34:12.419 align:middle line:90% This-- 00:34:12.419 --> 00:34:13.290 align:middle line:90% ANKIT MOHAN: This is-- 00:34:13.290 --> 00:34:14.100 align:middle line:90% STUDENT: Visible spectrum? 00:34:14.100 --> 00:34:15.017 align:middle line:90% ANKIT MOHAN: This is-- 00:34:15.017 --> 00:34:17.159 align:middle line:84% I think, it usually goes into near-IR at least, 00:34:17.159 --> 00:34:18.780 align:middle line:84% most of the remote sensing stuff. 00:34:18.780 --> 00:34:20.969 align:middle line:84% Because I think most of the interesting stuff 00:34:20.969 --> 00:34:24.870 align:middle line:84% of this kind of distinguishing thing actually happens in IR. 00:34:24.870 --> 00:34:27.078 align:middle line:84% But they do include visible light. 00:34:27.078 --> 00:34:29.370 align:middle line:84% PROFESSOR: They might have a band of 5 nanometers or 10 00:34:29.370 --> 00:34:36.449 align:middle line:84% nanometers and capture 20, 30 channels and [INAUDIBLE] 00:34:36.449 --> 00:34:40.163 align:middle line:84% high dimensional signal [INAUDIBLE].. 00:34:40.163 --> 00:34:41.580 align:middle line:84% ANKIT MOHAN: I'm not exactly sure. 00:34:41.580 --> 00:34:44.550 align:middle line:84% Do you know Michael, what trains they usually use for this kind 00:34:44.550 --> 00:34:45.610 align:middle line:90% of [INAUDIBLE]? 00:34:45.610 --> 00:34:47.610 align:middle line:84% MICHAEL: Usually-- I mean, I don't know usually, 00:34:47.610 --> 00:34:48.540 align:middle line:90% but they don't-- 00:34:48.540 --> 00:34:52.320 align:middle line:84% it is definitely common to go into the IR and sometimes 00:34:52.320 --> 00:34:53.760 align:middle line:90% [INAUDIBLE]. 00:34:53.760 --> 00:34:56.330 align:middle line:90% ANKIT MOHAN: Right. 00:34:56.330 --> 00:35:00.290 align:middle line:84% So, talking about UV, you can also do interesting photography 00:35:00.290 --> 00:35:02.270 align:middle line:90% in UV range. 00:35:02.270 --> 00:35:04.213 align:middle line:90% And this is an amazing website. 00:35:04.213 --> 00:35:06.380 align:middle line:84% If you get a chance, you should definitely visit it. 00:35:06.380 --> 00:35:08.150 align:middle line:84% They have-- this guy has like pictures 00:35:08.150 --> 00:35:11.930 align:middle line:84% of all kinds of flowers and both invisible spectrum 00:35:11.930 --> 00:35:12.830 align:middle line:90% and then in UV. 00:35:12.830 --> 00:35:14.810 align:middle line:84% And it turns out that the flowers 00:35:14.810 --> 00:35:17.840 align:middle line:84% look just amazingly different when you look at them in UV. 00:35:17.840 --> 00:35:21.680 align:middle line:84% And you have these-- almost these landing strips that 00:35:21.680 --> 00:35:25.430 align:middle line:84% invite the bees to come and sit-- 00:35:25.430 --> 00:35:26.180 align:middle line:90% give directions. 00:35:26.180 --> 00:35:28.602 align:middle line:84% Don't sit here, sit over here kind of almost. 00:35:28.602 --> 00:35:30.560 align:middle line:84% Whereas if you look at it in the visible light, 00:35:30.560 --> 00:35:33.200 align:middle line:84% it's all yellow or it's all red and there's hardly 00:35:33.200 --> 00:35:34.610 align:middle line:90% any difference between-- 00:35:34.610 --> 00:35:35.487 align:middle line:90% you don't see that-- 00:35:35.487 --> 00:35:36.320 align:middle line:90% [INTERPOSING VOICES] 00:35:36.320 --> 00:35:37.970 align:middle line:84% PROFESSOR: One is especially striking. 00:35:37.970 --> 00:35:38.600 align:middle line:90% ANKIT MOHAN: Yeah. 00:35:38.600 --> 00:35:40.517 align:middle line:84% Yeah, there's [INAUDIBLE] portions which are-- 00:35:40.517 --> 00:35:43.910 align:middle line:84% I think, well again, this is all fake color. 00:35:43.910 --> 00:35:45.950 align:middle line:84% There really is no color in this UV. 00:35:45.950 --> 00:35:48.770 align:middle line:90% 00:35:48.770 --> 00:35:52.940 align:middle line:84% But one thing I want to point out here is that you cannot do 00:35:52.940 --> 00:35:55.610 align:middle line:84% this type of photography with most traditional cameras 00:35:55.610 --> 00:35:59.180 align:middle line:84% because glass is actually-- it absorbs UV. 00:35:59.180 --> 00:36:03.020 align:middle line:84% So, they use these special rare Earth quartz lenses 00:36:03.020 --> 00:36:04.790 align:middle line:84% which are super, super expensive in order 00:36:04.790 --> 00:36:07.900 align:middle line:90% to do this kind of photography. 00:36:07.900 --> 00:36:12.610 align:middle line:90% 00:36:12.610 --> 00:36:15.760 align:middle line:90% OK, so-- 00:36:15.760 --> 00:36:20.140 align:middle line:84% PROFESSOR: What you could use, for example, what 00:36:20.140 --> 00:36:21.940 align:middle line:84% Professor [INAUDIBLE] was talking about 00:36:21.940 --> 00:36:27.250 align:middle line:84% instead of using a lens, you could use a [INAUDIBLE] zone 00:36:27.250 --> 00:36:32.380 align:middle line:84% plate which is like a pinhole camera except glorified. 00:36:32.380 --> 00:36:34.905 align:middle line:90% It has more interesting pattern. 00:36:34.905 --> 00:36:36.280 align:middle line:84% And then on the sensor, you could 00:36:36.280 --> 00:36:39.760 align:middle line:84% put a layer of fluorescent material, 00:36:39.760 --> 00:36:43.600 align:middle line:84% so that you will stimulate the fluorescent 00:36:43.600 --> 00:36:46.600 align:middle line:90% and then make an image for that. 00:36:46.600 --> 00:36:50.740 align:middle line:84% So, you can kind of go around some 00:36:50.740 --> 00:36:52.840 align:middle line:84% of the limitations in a traditional camera 00:36:52.840 --> 00:36:54.130 align:middle line:90% to do UV photography. 00:36:54.130 --> 00:36:56.080 align:middle line:84% That's what they do in a way for a lot 00:36:56.080 --> 00:37:00.280 align:middle line:90% of medical imaging [INAUDIBLE]. 00:37:00.280 --> 00:37:03.580 align:middle line:84% ANKIT MOHAN: Also a lot of these images that you find, they all 00:37:03.580 --> 00:37:04.720 align:middle line:90% use film photography. 00:37:04.720 --> 00:37:09.640 align:middle line:84% They rarely use digital cameras for these kind of things. 00:37:09.640 --> 00:37:12.480 align:middle line:84% I mean, even for near-IR, I think-- 00:37:12.480 --> 00:37:15.400 align:middle line:84% PROFESSOR: This [INAUDIBLE] is just one 00:37:15.400 --> 00:37:19.690 align:middle line:84% of the major craze that has been around for several years. 00:37:19.690 --> 00:37:21.760 align:middle line:84% So, remember the UV in the UV spectrum, 00:37:21.760 --> 00:37:24.190 align:middle line:84% these images are going to look just monochrome. 00:37:24.190 --> 00:37:25.510 align:middle line:90% It's black and white. 00:37:25.510 --> 00:37:27.730 align:middle line:84% But artists like to start assigning 00:37:27.730 --> 00:37:30.910 align:middle line:84% some colors to different intensity levels 00:37:30.910 --> 00:37:32.170 align:middle line:90% to seem more interesting. 00:37:32.170 --> 00:37:38.350 align:middle line:84% And for NASA and for astronomy, [INAUDIBLE] 00:37:38.350 --> 00:37:40.540 align:middle line:84% sometime over 20 years ago, [INAUDIBLE] 00:37:40.540 --> 00:37:42.760 align:middle line:84% put in those pictures of the nebulas 00:37:42.760 --> 00:37:44.710 align:middle line:84% and all that is sort of just putting them 00:37:44.710 --> 00:37:47.500 align:middle line:84% in [INAUDIBLE] really boring monochrome colors, 00:37:47.500 --> 00:37:49.330 align:middle line:90% let's start coloring them. 00:37:49.330 --> 00:37:54.960 align:middle line:84% Beautiful reddish hues and greenish lavas and [INAUDIBLE].. 00:37:54.960 --> 00:37:58.400 align:middle line:84% The real thing doesn't look like that. 00:37:58.400 --> 00:38:05.270 align:middle line:84% But it allows ordinary people to start appreciating astronomy. 00:38:05.270 --> 00:38:08.450 align:middle line:84% ANKIT MOHAN: Yeah, it's like the wallpaper on-- 00:38:08.450 --> 00:38:12.260 align:middle line:90% that comes with OS X has that. 00:38:12.260 --> 00:38:13.100 align:middle line:90% OK. 00:38:13.100 --> 00:38:17.480 align:middle line:84% So, now switching gears a little bit. 00:38:17.480 --> 00:38:21.073 align:middle line:84% This is more getting into the human perception of color. 00:38:21.073 --> 00:38:23.240 align:middle line:84% And I'm not going to get into too many details, just 00:38:23.240 --> 00:38:25.230 align:middle line:90% a very high level overview. 00:38:25.230 --> 00:38:28.610 align:middle line:84% So, many of you looked at this figure 00:38:28.610 --> 00:38:30.690 align:middle line:84% before, this chromaticity diagram. 00:38:30.690 --> 00:38:33.010 align:middle line:90% So, what this really-- 00:38:33.010 --> 00:38:37.010 align:middle line:84% this is the thing that people use in order 00:38:37.010 --> 00:38:39.870 align:middle line:84% to see how humans are actually sensing color. 00:38:39.870 --> 00:38:43.190 align:middle line:84% And what you have along over here 00:38:43.190 --> 00:38:50.900 align:middle line:84% is each of the spectrally pure colors, 00:38:50.900 --> 00:38:54.090 align:middle line:84% so went from 400 to 700 nanometers and everything 00:38:54.090 --> 00:38:54.590 align:middle line:90% inside. 00:38:54.590 --> 00:38:57.440 align:middle line:84% So, you have this blue over here going to green and then 00:38:57.440 --> 00:38:58.640 align:middle line:90% red over here. 00:38:58.640 --> 00:39:02.150 align:middle line:84% And anything that's inside is basically 00:39:02.150 --> 00:39:06.560 align:middle line:84% a combination of multiple of these colors 00:39:06.560 --> 00:39:08.460 align:middle line:90% that are on the outside. 00:39:08.460 --> 00:39:11.720 align:middle line:84% So, it turns out that most devices actually cannot-- 00:39:11.720 --> 00:39:14.420 align:middle line:84% so, this actually represents the space of all colors that 00:39:14.420 --> 00:39:16.520 align:middle line:90% the human eye can detect. 00:39:16.520 --> 00:39:18.560 align:middle line:84% It turns out that most devices actually 00:39:18.560 --> 00:39:22.220 align:middle line:84% occupy a much smaller space within this whole chromaticity 00:39:22.220 --> 00:39:25.140 align:middle line:84% diagram, this horseshoe shaped thing. 00:39:25.140 --> 00:39:29.695 align:middle line:84% And so sRGB which is the default color of most CRT monitors 00:39:29.695 --> 00:39:31.070 align:middle line:84% and it's the default color that's 00:39:31.070 --> 00:39:34.860 align:middle line:84% used on the internet is actually a much smaller region. 00:39:34.860 --> 00:39:36.920 align:middle line:84% You can only represent colors that 00:39:36.920 --> 00:39:40.010 align:middle line:84% are in between this triangle in sRGB. 00:39:40.010 --> 00:39:42.440 align:middle line:84% Now, what's interesting with this color space is that once 00:39:42.440 --> 00:39:46.490 align:middle line:84% you have these three points, any point 00:39:46.490 --> 00:39:48.380 align:middle line:84% that's inside this triangle can actually 00:39:48.380 --> 00:39:53.450 align:middle line:84% be represented by a sum of just these three color primaries. 00:39:53.450 --> 00:39:57.620 align:middle line:84% So, that's what I mean by fixed color primary is that most 00:39:57.620 --> 00:40:00.470 align:middle line:84% cameras that have a single green, single red, 00:40:00.470 --> 00:40:03.230 align:middle line:84% and single blue filter on the sensor 00:40:03.230 --> 00:40:06.067 align:middle line:84% actually have-- the green is somewhere over here, blue 00:40:06.067 --> 00:40:07.650 align:middle line:84% somewhere here, red is somewhere here. 00:40:07.650 --> 00:40:10.070 align:middle line:84% And that allows you to represent any color that 00:40:10.070 --> 00:40:13.140 align:middle line:90% lies within this triangle. 00:40:13.140 --> 00:40:17.360 align:middle line:84% Now, that's what the color response 00:40:17.360 --> 00:40:20.210 align:middle line:84% of traditional film, and that's what 00:40:20.210 --> 00:40:21.560 align:middle line:90% it looks like for most cameras. 00:40:21.560 --> 00:40:22.773 align:middle line:90% So, it's very similar. 00:40:22.773 --> 00:40:25.190 align:middle line:84% There is very little difference between the color response 00:40:25.190 --> 00:40:25.850 align:middle line:90% of the two. 00:40:25.850 --> 00:40:28.160 align:middle line:84% And in both cases, the response is 00:40:28.160 --> 00:40:32.720 align:middle line:84% made to close the [INAUDIBLE] out of the human eye itself 00:40:32.720 --> 00:40:35.510 align:middle line:84% except for in firmware, like this firm [INAUDIBLE] 00:40:35.510 --> 00:40:38.540 align:middle line:84% I think is something that's very good for nature photography. 00:40:38.540 --> 00:40:42.710 align:middle line:84% So, they stress more on the red than what the digital camera 00:40:42.710 --> 00:40:44.360 align:middle line:90% does just because-- 00:40:44.360 --> 00:40:45.860 align:middle line:84% it's a little unnatural, the colors. 00:40:45.860 --> 00:40:48.830 align:middle line:84% PROFESSOR: [INAUDIBLE] for ocean photography. 00:40:48.830 --> 00:40:52.280 align:middle line:84% ANKIT MOHAN: It's any landscape kind of thing, sunrises 00:40:52.280 --> 00:40:54.650 align:middle line:90% and sunsets, nature. 00:40:54.650 --> 00:40:59.820 align:middle line:84% It doesn't work very well for skin tone, for example. 00:40:59.820 --> 00:41:01.880 align:middle line:84% So, this works fine as long as your color 00:41:01.880 --> 00:41:03.500 align:middle line:90% is within this triangle. 00:41:03.500 --> 00:41:06.230 align:middle line:84% But once you want to represent a color that's outside, 00:41:06.230 --> 00:41:07.580 align:middle line:90% it becomes-- 00:41:07.580 --> 00:41:11.390 align:middle line:84% it's not possible to do that because this RGB, the values, 00:41:11.390 --> 00:41:13.370 align:middle line:90% can only be between 0 and 1. 00:41:13.370 --> 00:41:15.410 align:middle line:84% So, there are a number of algorithms 00:41:15.410 --> 00:41:17.150 align:middle line:84% that you can use in order to estimate 00:41:17.150 --> 00:41:18.470 align:middle line:90% what the color should be. 00:41:18.470 --> 00:41:22.250 align:middle line:84% But each of them is a [INAUDIBLE] lose information 00:41:22.250 --> 00:41:24.680 align:middle line:84% because you may project it to the nearest point 00:41:24.680 --> 00:41:28.670 align:middle line:84% or you may project it to the perceptually nearest point 00:41:28.670 --> 00:41:29.990 align:middle line:90% and all those kinds of things. 00:41:29.990 --> 00:41:32.750 align:middle line:84% But eventually, you end up losing this information. 00:41:32.750 --> 00:41:35.600 align:middle line:84% So, it turns out like all this region over here, 00:41:35.600 --> 00:41:37.550 align:middle line:84% it's very hard to represent colors over here 00:41:37.550 --> 00:41:39.860 align:middle line:84% just because you are triangle is actually-- 00:41:39.860 --> 00:41:41.090 align:middle line:90% it's outside this triangle. 00:41:41.090 --> 00:41:43.950 align:middle line:90% 00:41:43.950 --> 00:41:49.040 align:middle line:84% So, one alternative you can do is that instead of using-- 00:41:49.040 --> 00:41:50.930 align:middle line:84% putting the color primary over here, 00:41:50.930 --> 00:41:53.250 align:middle line:84% you can put a color primary out there. 00:41:53.250 --> 00:41:55.520 align:middle line:84% So, now you have this really big color gamut 00:41:55.520 --> 00:41:57.140 align:middle line:84% and you can represent all colors that 00:41:57.140 --> 00:41:58.970 align:middle line:90% lie inside this color space. 00:41:58.970 --> 00:42:00.710 align:middle line:84% Unfortunately, what that means is 00:42:00.710 --> 00:42:02.930 align:middle line:84% that your green is now very close to, 00:42:02.930 --> 00:42:04.580 align:middle line:90% let's say 520 nanometers here. 00:42:04.580 --> 00:42:06.140 align:middle line:84% So, it's a very spectrally pure-- 00:42:06.140 --> 00:42:09.410 align:middle line:84% it's just one very narrow range of wavelengths 00:42:09.410 --> 00:42:10.680 align:middle line:90% like that over there. 00:42:10.680 --> 00:42:15.440 align:middle line:84% And that means you have to use a very sharp wavelength profile, 00:42:15.440 --> 00:42:18.800 align:middle line:84% LED, or a laser or something of that illuminating it 00:42:18.800 --> 00:42:21.740 align:middle line:90% or a filter that's really dark. 00:42:21.740 --> 00:42:26.630 align:middle line:84% So, it turns out that these optimal color 00:42:26.630 --> 00:42:29.420 align:middle line:84% gamuts that you have there are very good compromise 00:42:29.420 --> 00:42:31.880 align:middle line:84% between having a wide gamut and having filters 00:42:31.880 --> 00:42:36.360 align:middle line:84% that allow a large amount of power to come into the system. 00:42:36.360 --> 00:42:42.140 align:middle line:84% So, if you have these primaries very close to the edges, 00:42:42.140 --> 00:42:44.220 align:middle line:84% you end up throwing away too much of the light. 00:42:44.220 --> 00:42:47.880 align:middle line:84% You get very little light coming in. 00:42:47.880 --> 00:42:49.370 align:middle line:84% So, what you'd ideally want to do 00:42:49.370 --> 00:42:52.400 align:middle line:84% is have these adaptive color primaries. 00:42:52.400 --> 00:42:54.830 align:middle line:84% So, for a scene like this, you don't have too many reds 00:42:54.830 --> 00:42:56.090 align:middle line:90% in this for example. 00:42:56.090 --> 00:42:58.580 align:middle line:84% You want color primaries to be like this that's 00:42:58.580 --> 00:43:00.270 align:middle line:90% just some arbitrary shape. 00:43:00.270 --> 00:43:03.097 align:middle line:84% And for a scene like this, you want them to be more like that. 00:43:03.097 --> 00:43:05.180 align:middle line:84% Rather than have the same set of colored primaries 00:43:05.180 --> 00:43:07.430 align:middle line:84% for every image that you capture, you want-- 00:43:07.430 --> 00:43:09.680 align:middle line:84% you want to be a little more adaptive about it. 00:43:09.680 --> 00:43:12.870 align:middle line:90% STUDENT: [INAUDIBLE] 00:43:12.870 --> 00:43:16.100 align:middle line:84% ANKIT MOHAN: So, it's kind of like the equalizer, you mean? 00:43:16.100 --> 00:43:20.060 align:middle line:84% So, you want to be able to tweak the various wavelengths-- what 00:43:20.060 --> 00:43:21.530 align:middle line:84% wavelengths should be sensed more 00:43:21.530 --> 00:43:23.270 align:middle line:90% and what should be sensed less. 00:43:23.270 --> 00:43:25.400 align:middle line:84% PROFESSOR: And if you're playing pop music 00:43:25.400 --> 00:43:27.560 align:middle line:84% versus classical music, you may want 00:43:27.560 --> 00:43:31.890 align:middle line:84% to change your synthesizer, more bass, less treble, and so on. 00:43:31.890 --> 00:43:37.880 align:middle line:84% And because those settings on your equalizer 00:43:37.880 --> 00:43:40.120 align:middle line:84% are also boosting certain frequencies 00:43:40.120 --> 00:43:44.390 align:middle line:84% and attenuating other frequencies. 00:43:44.390 --> 00:43:46.880 align:middle line:84% In case of cameras, the RGB has already 00:43:46.880 --> 00:43:49.760 align:middle line:84% been fixed [INAUDIBLE] three points. 00:43:49.760 --> 00:43:51.860 align:middle line:84% And those three points have been fixed. 00:43:51.860 --> 00:43:55.020 align:middle line:84% But maybe you ought to change the frequency. 00:43:55.020 --> 00:43:58.433 align:middle line:90% 00:43:58.433 --> 00:44:01.100 align:middle line:84% ANKIT MOHAN: So, that's what I'm going to briefly talk about how 00:44:01.100 --> 00:44:02.300 align:middle line:90% one way of doing that. 00:44:02.300 --> 00:44:05.420 align:middle line:90% And-- is everyone aware? 00:44:05.420 --> 00:44:07.685 align:middle line:84% Have you talked about [INAUDIBLE]?? 00:44:07.685 --> 00:44:09.000 align:middle line:90% STUDENT: I have a question. 00:44:09.000 --> 00:44:12.920 align:middle line:84% So, here in this graph, the chromaticity diagram, 00:44:12.920 --> 00:44:16.280 align:middle line:84% it seems like you can represent it in a 2D plane 00:44:16.280 --> 00:44:19.550 align:middle line:84% so which would mean that two primaries are 00:44:19.550 --> 00:44:23.600 align:middle line:84% enough to define any color, because it's just a 2D plane. 00:44:23.600 --> 00:44:24.670 align:middle line:90% [INTERPOSING VOICES] 00:44:24.670 --> 00:44:26.642 align:middle line:84% ANKIT MOHAN: There's an intensity, which is-- 00:44:26.642 --> 00:44:28.100 align:middle line:84% this is a projection of the whole-- 00:44:28.100 --> 00:44:29.990 align:middle line:84% STUDENT: Oh, so it's actually a 3D thing. 00:44:29.990 --> 00:44:32.090 align:middle line:90% [INTERPOSING VOICES] 00:44:32.090 --> 00:44:34.640 align:middle line:84% PROFESSOR: But in terms of the color, you're right. 00:44:34.640 --> 00:44:36.140 align:middle line:84% You could represent with two numbers 00:44:36.140 --> 00:44:37.820 align:middle line:84% and the third one is the intensity. 00:44:37.820 --> 00:44:41.840 align:middle line:84% And that's why you have LUV or LAB or what is luminance 00:44:41.840 --> 00:44:44.450 align:middle line:84% and there's A and B which is chromaticity 00:44:44.450 --> 00:44:51.470 align:middle line:84% A and Chromaticity B or [INAUDIBLE] is the old one. 00:44:51.470 --> 00:44:53.840 align:middle line:84% ANKIT MOHAN: One of the things that these-- 00:44:53.840 --> 00:44:57.053 align:middle line:84% so this-- when I said this is the RBG color space, 00:44:57.053 --> 00:44:58.220 align:middle line:90% this is actually incomplete. 00:44:58.220 --> 00:44:59.950 align:middle line:90% Just defining RGB is not enough. 00:44:59.950 --> 00:45:01.700 align:middle line:84% You need to define the white point inside, 00:45:01.700 --> 00:45:02.960 align:middle line:90% where is the white. 00:45:02.960 --> 00:45:06.360 align:middle line:84% And that's also something that's defined over there. 00:45:06.360 --> 00:45:08.510 align:middle line:84% And that-- if you look at the third dimension, 00:45:08.510 --> 00:45:11.390 align:middle line:84% that's what represent the various levels of gray. 00:45:11.390 --> 00:45:13.430 align:middle line:84% So, in order to represent the intensity, 00:45:13.430 --> 00:45:15.110 align:middle line:90% you need that third axis. 00:45:15.110 --> 00:45:18.100 align:middle line:84% MICHAEL: And let me emphasize something that it's easier 00:45:18.100 --> 00:45:20.300 align:middle line:90% to lose I think in this. 00:45:20.300 --> 00:45:23.360 align:middle line:84% We tend to use the words color and wavelength interchangeably 00:45:23.360 --> 00:45:24.290 align:middle line:90% in some situations. 00:45:24.290 --> 00:45:29.040 align:middle line:84% And wavelength is a truly physical phenomenon. 00:45:29.040 --> 00:45:32.630 align:middle line:84% It's a property universe of the light itself. 00:45:32.630 --> 00:45:37.770 align:middle line:84% Color is specific to human vision, maybe other animals 00:45:37.770 --> 00:45:38.270 align:middle line:90% too. 00:45:38.270 --> 00:45:40.790 align:middle line:84% But it's different for them than it is for us. 00:45:40.790 --> 00:45:42.500 align:middle line:84% So, when we talk about turning wavelength 00:45:42.500 --> 00:45:44.000 align:middle line:84% into color, what we're talking about 00:45:44.000 --> 00:45:46.730 align:middle line:84% is the process of human perception. 00:45:46.730 --> 00:45:51.530 align:middle line:84% And so this plot, for example, is specific to human vision. 00:45:51.530 --> 00:45:55.640 align:middle line:84% If it's for some other creature or some physical device 00:45:55.640 --> 00:45:59.210 align:middle line:84% that we built, it would have a completely different plot. 00:45:59.210 --> 00:46:00.900 align:middle line:90% PROFESSOR: Exactly. 00:46:00.900 --> 00:46:04.710 align:middle line:84% So here's-- before we get onto the color sensing, 00:46:04.710 --> 00:46:06.630 align:middle line:90% here's an interesting puzzle. 00:46:06.630 --> 00:46:10.740 align:middle line:84% If you take a red laser pointer, which-- 00:46:10.740 --> 00:46:12.870 align:middle line:84% they're using a red color for right now. 00:46:12.870 --> 00:46:15.480 align:middle line:84% And as Mike just reminded us, don't 00:46:15.480 --> 00:46:17.372 align:middle line:84% think of color, think of the wavelength. 00:46:17.372 --> 00:46:19.080 align:middle line:84% So, let's say it has certain wavelength-- 00:46:19.080 --> 00:46:20.030 align:middle line:90% I don't know-- 00:46:20.030 --> 00:46:22.213 align:middle line:84% 680 nanometers, something like that? 00:46:22.213 --> 00:46:23.130 align:middle line:90% MICHAEL: 630 probably. 00:46:23.130 --> 00:46:25.530 align:middle line:90% PROFESSOR: 630. 00:46:25.530 --> 00:46:27.420 align:middle line:84% And if I just take the laser pointer 00:46:27.420 --> 00:46:32.250 align:middle line:84% and shine it in a piece of water-- a piece of glass 00:46:32.250 --> 00:46:38.110 align:middle line:84% or a tank of water, what's going to happen to it? 00:46:38.110 --> 00:46:40.710 align:middle line:90% [INAUDIBLE] 00:46:40.710 --> 00:46:43.380 align:middle line:90% [INAUDIBLE] shine laser in it. 00:46:43.380 --> 00:46:44.980 align:middle line:90% What happens to it? 00:46:44.980 --> 00:46:46.650 align:middle line:90% [INTERPOSING VOICES] 00:46:46.650 --> 00:46:49.260 align:middle line:90% It bends. 00:46:49.260 --> 00:46:53.580 align:middle line:90% And why does it bend? 00:46:53.580 --> 00:46:55.920 align:middle line:84% Does anything change about its physical properties? 00:46:55.920 --> 00:46:58.740 align:middle line:90% 00:46:58.740 --> 00:47:01.260 align:middle line:90% Of course, it does. 00:47:01.260 --> 00:47:06.180 align:middle line:84% Either the wavelength or speed or something has to change. 00:47:06.180 --> 00:47:08.800 align:middle line:84% The speed is actually decreasing. 00:47:08.800 --> 00:47:16.065 align:middle line:84% So c in air versus c in water is related by a factor of what? 00:47:16.065 --> 00:47:17.240 align:middle line:90% STUDENT: [INAUDIBLE] 00:47:17.240 --> 00:47:19.860 align:middle line:90% PROFESSOR: The refractive index. 00:47:19.860 --> 00:47:27.550 align:middle line:84% So, this is actually reduced by a factor of 2/3. 00:47:27.550 --> 00:47:31.020 align:middle line:84% Now, we know that here, we have a wavelength of light 00:47:31.020 --> 00:47:34.350 align:middle line:90% times the frequency of light. 00:47:34.350 --> 00:47:36.180 align:middle line:84% I'm just talking about a for air. 00:47:36.180 --> 00:47:37.920 align:middle line:90% This is for a laser. 00:47:37.920 --> 00:47:41.445 align:middle line:84% And similarly here, we have wavelength in water 00:47:41.445 --> 00:47:44.790 align:middle line:90% and frequency in water. 00:47:44.790 --> 00:47:49.020 align:middle line:90% So, this has gone down by 2/3. 00:47:49.020 --> 00:47:53.405 align:middle line:84% Something here also has to go down by a factor of 2/3. 00:47:53.405 --> 00:47:53.905 align:middle line:90% Yes. 00:47:53.905 --> 00:47:55.543 align:middle line:84% STUDENT: [INAUDIBLE] frequencies fixed. 00:47:55.543 --> 00:47:57.460 align:middle line:84% ANKIT MOHAN: Yeah, the frequency [INAUDIBLE].. 00:47:57.460 --> 00:47:59.800 align:middle line:84% PROFESSOR: So, the wavelength has changed. 00:47:59.800 --> 00:48:04.630 align:middle line:90% So, that means from 630 you say? 00:48:04.630 --> 00:48:08.210 align:middle line:84% From 630, we have gone down to 2/3 of that. 00:48:08.210 --> 00:48:14.440 align:middle line:90% So, that would be what 420. 00:48:14.440 --> 00:48:16.780 align:middle line:84% So now, this red laser is actually-- 00:48:16.780 --> 00:48:19.220 align:middle line:84% has a wavelength of 420 nanometers. 00:48:19.220 --> 00:48:22.330 align:middle line:84% And so we started from somewhere here 00:48:22.330 --> 00:48:27.670 align:middle line:84% and the light wavelength is actually now even further down. 00:48:27.670 --> 00:48:31.870 align:middle line:84% So does it mean that when you shine a laser in water, 00:48:31.870 --> 00:48:35.650 align:middle line:84% it's actually blue because it's wavelength now 00:48:35.650 --> 00:48:37.074 align:middle line:90% is 420 nanometers? 00:48:37.074 --> 00:48:39.910 align:middle line:90% 00:48:39.910 --> 00:48:40.600 align:middle line:90% Is it blue? 00:48:40.600 --> 00:48:42.700 align:middle line:84% STUDENT: Well, we can't ever see that because we 00:48:42.700 --> 00:48:44.380 align:middle line:90% are [INAUDIBLE]. 00:48:44.380 --> 00:48:47.570 align:middle line:90% 00:48:47.570 --> 00:48:52.977 align:middle line:84% PROFESSOR: So, when it comes out it's back to 630. 00:48:52.977 --> 00:48:53.810 align:middle line:90% [INTERPOSING VOICES] 00:48:53.810 --> 00:48:55.570 align:middle line:90% But when it's inside, it's 420. 00:48:55.570 --> 00:49:01.270 align:middle line:84% MICHAEL: You could immerse the CCD in water, for example. 00:49:01.270 --> 00:49:04.305 align:middle line:84% ANKIT MOHAN: I guess we sense frequency more than wavelength. 00:49:04.305 --> 00:49:05.180 align:middle line:90% PROFESSOR: Very good. 00:49:05.180 --> 00:49:07.840 align:middle line:90% [INAUDIBLE] 00:49:07.840 --> 00:49:10.000 align:middle line:90% So, remember that message. 00:49:10.000 --> 00:49:14.320 align:middle line:84% Don't think about color, think about the physical properties. 00:49:14.320 --> 00:49:16.960 align:middle line:84% And the way to think about that is you 00:49:16.960 --> 00:49:22.270 align:middle line:84% may have different wavelengths but they mean different things 00:49:22.270 --> 00:49:23.860 align:middle line:90% in different media. 00:49:23.860 --> 00:49:28.180 align:middle line:84% In air, if you really want to think about colors, in air, 00:49:28.180 --> 00:49:30.010 align:middle line:90% 630 is red. 00:49:30.010 --> 00:49:33.790 align:middle line:90% In water, 420 is red. 00:49:33.790 --> 00:49:35.300 align:middle line:90% So, it gets too confusing. 00:49:35.300 --> 00:49:37.720 align:middle line:84% So, you can use colors when they're convenient. 00:49:37.720 --> 00:49:40.090 align:middle line:84% But when we start talking about physical interaction, 00:49:40.090 --> 00:49:43.150 align:middle line:84% it's better to talk about wavelengths or really something 00:49:43.150 --> 00:49:45.923 align:middle line:84% that remains constant, [INAUDIBLE].. 00:49:45.923 --> 00:49:47.590 align:middle line:84% ANKIT MOHAN: I think the important thing 00:49:47.590 --> 00:49:51.200 align:middle line:84% is that the energy of the ray remains the same. 00:49:51.200 --> 00:49:53.200 align:middle line:90% It's e is equal to hnu-- 00:49:53.200 --> 00:49:54.100 align:middle line:90% it's the frequency. 00:49:54.100 --> 00:49:57.100 align:middle line:84% So, that's what the [INAUDIBLE] it's 00:49:57.100 --> 00:50:00.400 align:middle line:84% the energy of the electrons that get displaced. 00:50:00.400 --> 00:50:04.840 align:middle line:84% But when-- so, that's why I said for whatever reasons when 00:50:04.840 --> 00:50:06.400 align:middle line:84% you're talking about visible light, 00:50:06.400 --> 00:50:09.010 align:middle line:84% they always like to talk about it with wavelength. 00:50:09.010 --> 00:50:12.430 align:middle line:84% And it's not entirely correct because as Ramesh pointed out, 00:50:12.430 --> 00:50:16.420 align:middle line:84% this figure is only true for air. 00:50:16.420 --> 00:50:19.630 align:middle line:84% And it won't be-- it would be completely very different 00:50:19.630 --> 00:50:20.770 align:middle line:90% if you were underwater. 00:50:20.770 --> 00:50:23.080 align:middle line:84% But it's just a convention that people like to follow. 00:50:23.080 --> 00:50:25.498 align:middle line:84% And that's what we are sticking too. 00:50:25.498 --> 00:50:27.290 align:middle line:84% PROFESSOR: [INAUDIBLE] to ask this question 00:50:27.290 --> 00:50:30.470 align:middle line:84% to your photographer friends because they 00:50:30.470 --> 00:50:33.270 align:middle line:90% like to talk about colors. 00:50:33.270 --> 00:50:36.200 align:middle line:84% ANKIT MOHAN: So again, going back over here. 00:50:36.200 --> 00:50:39.003 align:middle line:90% As I was saying, in different-- 00:50:39.003 --> 00:50:41.420 align:middle line:84% that's why they have both the frequency and the wavelength 00:50:41.420 --> 00:50:42.080 align:middle line:90% over here. 00:50:42.080 --> 00:50:46.550 align:middle line:84% In different fields and in different-- 00:50:46.550 --> 00:50:48.817 align:middle line:84% for different purposes, you would use either frequency 00:50:48.817 --> 00:50:49.400 align:middle line:90% or wavelength. 00:50:49.400 --> 00:50:51.500 align:middle line:84% It's just that for photography and for especially 00:50:51.500 --> 00:50:54.930 align:middle line:84% for this kind of imaging and [INAUDIBLE] the visible region, 00:50:54.930 --> 00:50:57.320 align:middle line:84% it's something that people usually use wavelength and not 00:50:57.320 --> 00:50:58.490 align:middle line:90% the frequency or the energy. 00:50:58.490 --> 00:51:01.310 align:middle line:90% 00:51:01.310 --> 00:51:03.905 align:middle line:84% So, we wanted to try to come up with a way of having 00:51:03.905 --> 00:51:05.900 align:middle line:90% adaptive color primaries. 00:51:05.900 --> 00:51:10.910 align:middle line:84% And so I'm going to go through a analysis of an optical system 00:51:10.910 --> 00:51:11.690 align:middle line:90% that we developed. 00:51:11.690 --> 00:51:14.060 align:middle line:84% And it's going-- everything that I'm 00:51:14.060 --> 00:51:16.670 align:middle line:84% going to discuss is going to have be in what's 00:51:16.670 --> 00:51:18.240 align:middle line:90% called a flashland case. 00:51:18.240 --> 00:51:22.940 align:middle line:84% So, it's just in two dimensions but it scales up 00:51:22.940 --> 00:51:25.400 align:middle line:90% to a real 3D or 4D case also. 00:51:25.400 --> 00:51:27.560 align:middle line:84% So, we start with a simple 1D signal 00:51:27.560 --> 00:51:30.140 align:middle line:84% and this 1D signal is arbitrary intensity. 00:51:30.140 --> 00:51:34.520 align:middle line:84% And along-- this is the x position of the signal 00:51:34.520 --> 00:51:35.840 align:middle line:90% and this is the intensity. 00:51:35.840 --> 00:51:37.670 align:middle line:84% And it's a white signal, which means 00:51:37.670 --> 00:51:40.630 align:middle line:84% it has all the visible frequencies between 400 00:51:40.630 --> 00:51:41.990 align:middle line:90% and 700 nanometers. 00:51:41.990 --> 00:51:45.050 align:middle line:84% And the intensity of each one of those wavelengths 00:51:45.050 --> 00:51:47.130 align:middle line:90% is actually the same. 00:51:47.130 --> 00:51:49.587 align:middle line:84% So, that's what the wavelength profile 00:51:49.587 --> 00:51:51.170 align:middle line:84% or the color profile, spectral profile 00:51:51.170 --> 00:51:54.020 align:middle line:84% looks like at any point between a and c. 00:51:54.020 --> 00:51:58.700 align:middle line:84% So, we take this signal and we put it in front of a pinhole. 00:51:58.700 --> 00:52:01.070 align:middle line:84% So, here's your signal and here's a pinhole. 00:52:01.070 --> 00:52:04.040 align:middle line:84% So, the pinhole essentially creates an inverted image 00:52:04.040 --> 00:52:04.730 align:middle line:90% of the signal. 00:52:04.730 --> 00:52:08.075 align:middle line:84% So, you have this a,b, c here, you get a prime, b prime, 00:52:08.075 --> 00:52:08.900 align:middle line:90% c prime over here. 00:52:08.900 --> 00:52:10.430 align:middle line:90% It's just a pinhole camera. 00:52:10.430 --> 00:52:16.460 align:middle line:84% So now, instead of positioning a film or sensor over here, 00:52:16.460 --> 00:52:18.080 align:middle line:90% we put a lens in this plane. 00:52:18.080 --> 00:52:20.000 align:middle line:84% And this lens essentially [? complements ?] 00:52:20.000 --> 00:52:21.770 align:middle line:84% all these rays, so you have a prime, 00:52:21.770 --> 00:52:25.640 align:middle line:84% b prime, c prime, just like an orthographic set up so-- 00:52:25.640 --> 00:52:27.830 align:middle line:84% except it's inverted which is not so important, 00:52:27.830 --> 00:52:33.120 align:middle line:84% but you have a ray coming in for each point in the scene. 00:52:33.120 --> 00:52:35.840 align:middle line:84% So, next we place a prism in front of this. 00:52:35.840 --> 00:52:38.450 align:middle line:84% And now as I talked about-- mentioned earlier 00:52:38.450 --> 00:52:41.210 align:middle line:84% that a prism actually bends the incoming ray 00:52:41.210 --> 00:52:43.100 align:middle line:84% where the bending angle actually depends 00:52:43.100 --> 00:52:45.240 align:middle line:90% on the wavelength of light. 00:52:45.240 --> 00:52:47.690 align:middle line:84% So, what I'm going to show for simplicity 00:52:47.690 --> 00:52:51.920 align:middle line:84% over here is that when you have a white light coming in, 00:52:51.920 --> 00:52:52.980 align:middle line:90% the green-- 00:52:52.980 --> 00:52:55.160 align:middle line:84% the red corresponding to the green wavelength 00:52:55.160 --> 00:52:57.710 align:middle line:90% are 550 nanometers or something. 00:52:57.710 --> 00:53:00.350 align:middle line:84% Actually goes through straight and bend. 00:53:00.350 --> 00:53:04.220 align:middle line:84% And red gets bent upwards and blue gets bent downwards. 00:53:04.220 --> 00:53:06.698 align:middle line:84% Now, once again, I'm going against what 00:53:06.698 --> 00:53:07.490 align:middle line:90% Michael was saying. 00:53:07.490 --> 00:53:08.990 align:middle line:84% I'm calling it red, green, and blue, 00:53:08.990 --> 00:53:13.012 align:middle line:84% but it really is the wavelength, 400, 500, and 700 nanometers. 00:53:13.012 --> 00:53:15.470 align:middle line:84% I'm just going to call red, green, blue because it's easier 00:53:15.470 --> 00:53:17.280 align:middle line:90% to talk with it that way. 00:53:17.280 --> 00:53:19.340 align:middle line:84% And the other thing to notice here 00:53:19.340 --> 00:53:21.200 align:middle line:84% is that I'm only drawing these three rays, 00:53:21.200 --> 00:53:24.373 align:middle line:84% but really, it's a whole fan of rays because it's a continuum 00:53:24.373 --> 00:53:25.415 align:middle line:90% and it's a white object-- 00:53:25.415 --> 00:53:26.957 align:middle line:84% PROFESSOR: A whole rainbow basically. 00:53:26.957 --> 00:53:29.060 align:middle line:84% ANKIT MOHAN: You have the whole rainbow here. 00:53:29.060 --> 00:53:31.010 align:middle line:84% And I'm just drawing three of those rays. 00:53:31.010 --> 00:53:33.350 align:middle line:84% So, you would actually have a ray going down here 00:53:33.350 --> 00:53:35.630 align:middle line:84% and up there and anywhere between this blue 00:53:35.630 --> 00:53:38.330 align:middle line:90% and the red ray that we have. 00:53:38.330 --> 00:53:43.530 align:middle line:84% So, now looking at this prism more closely, along this axis, 00:53:43.530 --> 00:53:46.320 align:middle line:84% we have the spatial points of the scene. 00:53:46.320 --> 00:53:48.380 align:middle line:84% So, you have a prime, b prime, and c prime. 00:53:48.380 --> 00:53:50.840 align:middle line:84% And coming out of each of those points, 00:53:50.840 --> 00:53:54.560 align:middle line:84% you have this wavelength angle or lambda. 00:53:54.560 --> 00:53:58.220 align:middle line:84% Now, this figure should remind you of something. 00:53:58.220 --> 00:54:03.330 align:middle line:90% And anyone wants to guess? 00:54:03.330 --> 00:54:05.270 align:middle line:84% Does this figure look familiar to-- 00:54:05.270 --> 00:54:09.060 align:middle line:90% [INTERPOSING VOICES] 00:54:09.060 --> 00:54:11.990 align:middle line:84% It's exactly like a light field, except the only difference is 00:54:11.990 --> 00:54:14.540 align:middle line:90% that instead of having lambda-- 00:54:14.540 --> 00:54:17.828 align:middle line:84% theta over here or the angle, we replace it with lambda. 00:54:17.828 --> 00:54:20.120 align:middle line:84% So, this is something we call the spectral light field. 00:54:20.120 --> 00:54:24.020 align:middle line:84% It's-- the angular light field or the spatial light field-- 00:54:24.020 --> 00:54:25.100 align:middle line:90% the spectral light field. 00:54:25.100 --> 00:54:28.040 align:middle line:84% So, any point over here in x or lambda 00:54:28.040 --> 00:54:30.830 align:middle line:84% will represent the intensity of a ray 00:54:30.830 --> 00:54:34.460 align:middle line:84% in the space coming out of some point along a and c 00:54:34.460 --> 00:54:36.510 align:middle line:84% and going in a particular direction. 00:54:36.510 --> 00:54:40.820 align:middle line:84% And since we started with a white light source, 00:54:40.820 --> 00:54:43.940 align:middle line:84% it's going to have the same intensity along each 00:54:43.940 --> 00:54:48.300 align:middle line:90% of these along each wavelength. 00:54:48.300 --> 00:54:53.480 align:middle line:84% So, now-- since we've reduced this to nothing more than just 00:54:53.480 --> 00:54:55.070 align:middle line:84% a light field, it turns out we can 00:54:55.070 --> 00:54:57.380 align:middle line:84% use the various properties of a light field 00:54:57.380 --> 00:54:59.120 align:middle line:90% in analyzing the system. 00:54:59.120 --> 00:55:01.730 align:middle line:84% And the one-- again, this is something 00:55:01.730 --> 00:55:03.580 align:middle line:84% that you should be familiar with if you 00:55:03.580 --> 00:55:06.240 align:middle line:84% know what a light field is is that if you place 00:55:06.240 --> 00:55:08.660 align:middle line:84% a screen somewhere in front of a light field, 00:55:08.660 --> 00:55:11.720 align:middle line:84% you get a projection of the light field on that screen 00:55:11.720 --> 00:55:14.418 align:middle line:84% in a direction perpendicular to the screen in terms 00:55:14.418 --> 00:55:15.210 align:middle line:90% of the light field. 00:55:15.210 --> 00:55:18.860 align:middle line:84% So, what that means in the flashland cases 00:55:18.860 --> 00:55:20.960 align:middle line:90% that now if I place a light-- 00:55:20.960 --> 00:55:24.080 align:middle line:84% place a screen over here, this yellow thing, 00:55:24.080 --> 00:55:26.270 align:middle line:84% what that does to the light field is basically 00:55:26.270 --> 00:55:30.230 align:middle line:84% get a vertical projection, which is a direction perpendicular 00:55:30.230 --> 00:55:33.030 align:middle line:84% to the direction of the screen, which is along the x-axis. 00:55:33.030 --> 00:55:37.310 align:middle line:84% So, what we're getting for every point over here 00:55:37.310 --> 00:55:40.550 align:middle line:84% is an integration over the various all the wavelengths 00:55:40.550 --> 00:55:42.830 align:middle line:90% for that particular x position. 00:55:42.830 --> 00:55:44.780 align:middle line:84% And you essentially get the shape 00:55:44.780 --> 00:55:47.120 align:middle line:84% of the signal in this projection. 00:55:47.120 --> 00:55:49.550 align:middle line:84% So, if you were to place a screen over here, 00:55:49.550 --> 00:55:52.910 align:middle line:84% not surprisingly, you would get an image of the signal itself 00:55:52.910 --> 00:55:55.640 align:middle line:84% on the screen because it's so close to the prism it's-- 00:55:55.640 --> 00:55:58.130 align:middle line:84% these rays haven't dispersed as much yet. 00:55:58.130 --> 00:56:02.400 align:middle line:84% It's almost the same thing as if there was no prism over there. 00:56:02.400 --> 00:56:04.770 align:middle line:84% But now as you move the screen away 00:56:04.770 --> 00:56:08.350 align:middle line:84% from the prism, the angle of projection changes 00:56:08.350 --> 00:56:12.010 align:middle line:84% and it becomes more-- you get this sheer thing. 00:56:12.010 --> 00:56:16.590 align:middle line:84% And because of the sheer, you get the various wavelengths, 00:56:16.590 --> 00:56:18.960 align:middle line:84% the signal corresponding to the different wavelengths 00:56:18.960 --> 00:56:22.500 align:middle line:84% is now dispersed and it's overlapping and shifted. 00:56:22.500 --> 00:56:24.970 align:middle line:84% And once again, I've shown just the blue, green, 00:56:24.970 --> 00:56:26.100 align:middle line:90% and red wavelengths here. 00:56:26.100 --> 00:56:29.250 align:middle line:84% But clearly, it's a continuum and you're getting this rainbow 00:56:29.250 --> 00:56:31.140 align:middle line:90% like smear over here. 00:56:31.140 --> 00:56:33.900 align:middle line:84% And when you were looking through the diffraction 00:56:33.900 --> 00:56:35.580 align:middle line:84% grating, you could almost see this. 00:56:35.580 --> 00:56:38.280 align:middle line:84% You get a rainbow coming out of every point 00:56:38.280 --> 00:56:39.840 align:middle line:90% when you look through it. 00:56:39.840 --> 00:56:43.230 align:middle line:84% And that's similar to what you're getting over here. 00:56:43.230 --> 00:56:46.800 align:middle line:84% Now, if you move this plane away to infinity, 00:56:46.800 --> 00:56:48.450 align:middle line:84% when it's at infinity, you're going 00:56:48.450 --> 00:56:50.490 align:middle line:84% to get this horizontal projection 00:56:50.490 --> 00:56:52.410 align:middle line:90% through the spectral light. 00:56:52.410 --> 00:56:54.360 align:middle line:84% And now, for each point on the screen, 00:56:54.360 --> 00:56:57.360 align:middle line:84% we are integrating over all the spatial positions, 00:56:57.360 --> 00:57:02.020 align:middle line:84% all the points on the signal for each wavelength. 00:57:02.020 --> 00:57:05.040 align:middle line:84% So, what you get on the screen is nothing but a rainbow 00:57:05.040 --> 00:57:08.340 align:middle line:84% because we started with a white light source. 00:57:08.340 --> 00:57:13.420 align:middle line:84% And you have something going from blue to red. 00:57:13.420 --> 00:57:16.280 align:middle line:90% Any questions? 00:57:16.280 --> 00:57:19.870 align:middle line:84% So, what we've done is that we have this one plane where 00:57:19.870 --> 00:57:22.060 align:middle line:84% if we put a screen on that plane, 00:57:22.060 --> 00:57:25.630 align:middle line:84% we get the spectral characteristics of the signal 00:57:25.630 --> 00:57:27.788 align:middle line:90% that we started with. 00:57:27.788 --> 00:57:29.330 align:middle line:84% So, the problem with that, of course, 00:57:29.330 --> 00:57:31.300 align:middle line:84% is that this plane is infinitely far away. 00:57:31.300 --> 00:57:35.350 align:middle line:84% But we want to move it closer so we place another lens in front 00:57:35.350 --> 00:57:36.320 align:middle line:90% of the prism. 00:57:36.320 --> 00:57:38.030 align:middle line:90% And this lens does two things. 00:57:38.030 --> 00:57:40.900 align:middle line:84% The first is that it creates a copy of the prism itself 00:57:40.900 --> 00:57:44.830 align:middle line:84% on this plane at some distance from the lens 00:57:44.830 --> 00:57:47.998 align:middle line:84% where basically this is imaged, c prime 00:57:47.998 --> 00:57:49.540 align:middle line:84% is imaged at c double prime, b prime, 00:57:49.540 --> 00:57:51.020 align:middle line:90% and b double prime and so on. 00:57:51.020 --> 00:57:53.540 align:middle line:84% So, similar to the x that you had there, 00:57:53.540 --> 00:57:57.040 align:middle line:90% you get an x over here. 00:57:57.040 --> 00:58:00.280 align:middle line:84% But you also get this plane in the middle where 00:58:00.280 --> 00:58:03.580 align:middle line:84% each ray for each scene point of a particular wavelength 00:58:03.580 --> 00:58:06.950 align:middle line:84% actually converges and meets at a point on this plane. 00:58:06.950 --> 00:58:09.850 align:middle line:84% So, all the red rays are meeting here, 00:58:09.850 --> 00:58:11.590 align:middle line:84% green rays are meeting here, blue rays 00:58:11.590 --> 00:58:12.820 align:middle line:90% are meeting here and so on. 00:58:12.820 --> 00:58:17.010 align:middle line:90% 00:58:17.010 --> 00:58:19.100 align:middle line:84% So, once again, looking at this, when 00:58:19.100 --> 00:58:22.310 align:middle line:84% you place a screen at this plane at the end, which 00:58:22.310 --> 00:58:24.770 align:middle line:84% is an image of the prism itself, you're 00:58:24.770 --> 00:58:26.450 align:middle line:84% getting this vertical projection. 00:58:26.450 --> 00:58:29.460 align:middle line:84% And this is an image of the scene itself. 00:58:29.460 --> 00:58:31.250 align:middle line:84% This is where we place our sensor. 00:58:31.250 --> 00:58:34.830 align:middle line:84% And we call this the sensor plane of the optical system. 00:58:34.830 --> 00:58:38.420 align:middle line:84% And if you look at this plane, which 00:58:38.420 --> 00:58:41.360 align:middle line:84% had this nice cubed property of all the reds coming together, 00:58:41.360 --> 00:58:43.610 align:middle line:84% greens coming together, and so on, we actually 00:58:43.610 --> 00:58:45.320 align:middle line:90% get the horizontal projection. 00:58:45.320 --> 00:58:47.480 align:middle line:84% And this is the plane, which was infinitely before. 00:58:47.480 --> 00:58:49.500 align:middle line:84% It's moved much closer than that. 00:58:49.500 --> 00:58:51.970 align:middle line:84% And you get this nice looking rainbow at this plane. 00:58:51.970 --> 00:58:53.540 align:middle line:84% And we call this the rainbow plane. 00:58:53.540 --> 00:58:56.740 align:middle line:90% 00:58:56.740 --> 00:58:59.370 align:middle line:84% So, the nice cube property of the rainbow plane 00:58:59.370 --> 00:59:02.280 align:middle line:84% is that all the rays of a given wavelength coming 00:59:02.280 --> 00:59:04.290 align:middle line:84% from all the points in the scene actually 00:59:04.290 --> 00:59:06.840 align:middle line:84% converge to a single unique point in this plane. 00:59:06.840 --> 00:59:09.790 align:middle line:90% 00:59:09.790 --> 00:59:14.520 align:middle line:84% So, just to give you a couple of examples, 00:59:14.520 --> 00:59:17.940 align:middle line:84% if instead of the signal being wide if it had been completely 00:59:17.940 --> 00:59:21.750 align:middle line:84% red, let's say between 650 and 700 nanometers or something 00:59:21.750 --> 00:59:22.800 align:middle line:90% like that. 00:59:22.800 --> 00:59:25.020 align:middle line:84% When you took this horizontal projection 00:59:25.020 --> 00:59:27.360 align:middle line:84% at TR or the rainbow plane, you would only 00:59:27.360 --> 00:59:28.680 align:middle line:90% get this part of the rainbow. 00:59:28.680 --> 00:59:31.050 align:middle line:84% You won't get the other remaining rainbow. 00:59:31.050 --> 00:59:35.310 align:middle line:84% Similarly, if your signal was half blue and half red, 00:59:35.310 --> 00:59:38.460 align:middle line:84% you would get something like this at the rainbow plane. 00:59:38.460 --> 00:59:41.160 align:middle line:90% 00:59:41.160 --> 00:59:44.280 align:middle line:84% So, now if you place a mask in the rainbow plane, 00:59:44.280 --> 00:59:46.940 align:middle line:84% let's say you block out all these red rays that 00:59:46.940 --> 00:59:51.290 align:middle line:84% are going through, you actually put an occluder in that plane. 00:59:51.290 --> 00:59:53.840 align:middle line:84% When you-- what that essentially does 00:59:53.840 --> 00:59:55.910 align:middle line:84% is that it multiplies the incoming light 00:59:55.910 --> 01:00:00.260 align:middle line:84% field at this plane, which is this, with a light field which 01:00:00.260 --> 01:00:03.830 align:middle line:84% is all zeros corresponding to the wavelengths 01:00:03.830 --> 01:00:07.160 align:middle line:84% that you block and all ones corresponding to the all-- 01:00:07.160 --> 01:00:08.590 align:middle line:90% to all the other wavelengths. 01:00:08.590 --> 01:00:11.960 align:middle line:84% And so what you get is basically this-- 01:00:11.960 --> 01:00:13.680 align:middle line:90% what looks kind of like this. 01:00:13.680 --> 01:00:17.240 align:middle line:84% And now when you put the sensor at the screen 01:00:17.240 --> 01:00:19.700 align:middle line:84% at the sensor plane and you take the vertical projections, 01:00:19.700 --> 01:00:22.130 align:middle line:84% you don't get any of the red components. 01:00:22.130 --> 01:00:24.330 align:middle line:84% You only get the green and the blue component. 01:00:24.330 --> 01:00:26.660 align:middle line:84% So, you essentially-- by including 01:00:26.660 --> 01:00:30.493 align:middle line:84% the red channel over there or the red color 01:00:30.493 --> 01:00:31.910 align:middle line:84% or the red wavelengths over there, 01:00:31.910 --> 01:00:34.130 align:middle line:84% you remove them from the light field that 01:00:34.130 --> 01:00:35.720 align:middle line:90% gets projected on the sensor. 01:00:35.720 --> 01:00:38.420 align:middle line:84% And you only get what looks like cyan. 01:00:38.420 --> 01:00:41.510 align:middle line:90% It's just green plus blue. 01:00:41.510 --> 01:00:44.150 align:middle line:84% If you were to do that to the green channel, 01:00:44.150 --> 01:00:47.270 align:middle line:84% you could just occlude that you have a big 0 over here 01:00:47.270 --> 01:00:50.580 align:middle line:84% and you only get this plus this and this plus this 01:00:50.580 --> 01:00:54.090 align:middle line:84% and so it looks magenta and so on. 01:00:54.090 --> 01:00:57.230 align:middle line:84% So, essentially what you can do is place any arbitrary mask 01:00:57.230 --> 01:00:59.360 align:middle line:84% you want in this rainbow plane and that 01:00:59.360 --> 01:01:03.630 align:middle line:84% will influence what colors get sensed by the sensor 01:01:03.630 --> 01:01:06.560 align:middle line:84% and what colors are not sensed by the sensor. 01:01:06.560 --> 01:01:10.280 align:middle line:84% So, it-- by placing a mask in the rainbow plane, 01:01:10.280 --> 01:01:12.680 align:middle line:84% it allows you to control effectively 01:01:12.680 --> 01:01:16.880 align:middle line:84% the spectral sensitivity of the whole imaging system 01:01:16.880 --> 01:01:20.510 align:middle line:84% on the camera by modulating what rays go through 01:01:20.510 --> 01:01:24.680 align:middle line:84% and what rays get occluded over there. 01:01:24.680 --> 01:01:27.730 align:middle line:84% So, that's what the whole optical system looks like. 01:01:27.730 --> 01:01:31.870 align:middle line:84% We have the lens, the pinhole, the prism, another lens 01:01:31.870 --> 01:01:36.280 align:middle line:84% that images the rainbow plane and we place a mask over there 01:01:36.280 --> 01:01:37.800 align:middle line:84% and then the image sensor itself. 01:01:37.800 --> 01:01:40.180 align:middle line:84% PROFESSOR: So what's the benefit of this 01:01:40.180 --> 01:01:43.577 align:middle line:84% and what's the disadvantage compared to the other schemes 01:01:43.577 --> 01:01:44.410 align:middle line:90% we have seen before? 01:01:44.410 --> 01:01:47.710 align:middle line:90% 01:01:47.710 --> 01:01:49.960 align:middle line:84% Let's look at the disadvantage [INAUDIBLE].. 01:01:49.960 --> 01:01:51.760 align:middle line:84% STUDENT: Well, starting with the pinhole. 01:01:51.760 --> 01:01:52.968 align:middle line:90% PROFESSOR: That's a bad idea. 01:01:52.968 --> 01:01:54.792 align:middle line:90% [INTERPOSING VOICES] 01:01:54.792 --> 01:01:55.750 align:middle line:90% It's always a bad idea. 01:01:55.750 --> 01:01:58.030 align:middle line:84% [INAUDIBLE] pinhole when you're blocking light, 01:01:58.030 --> 01:02:00.380 align:middle line:90% it's always a bad idea. 01:02:00.380 --> 01:02:01.560 align:middle line:90% So, what would be-- 01:02:01.560 --> 01:02:02.560 align:middle line:90% does everybody see that? 01:02:02.560 --> 01:02:07.675 align:middle line:84% There's a-- what would happen if the pinhole was made larger? 01:02:07.675 --> 01:02:10.902 align:middle line:90% 01:02:10.902 --> 01:02:12.610 align:middle line:84% STUDENT: When you have to deal with focus 01:02:12.610 --> 01:02:15.008 align:middle line:90% problems [INAUDIBLE]. 01:02:15.008 --> 01:02:16.300 align:middle line:90% PROFESSOR: So, you'll get blur. 01:02:16.300 --> 01:02:17.758 align:middle line:84% But what kind of blur [INAUDIBLE]?? 01:02:17.758 --> 01:02:21.280 align:middle line:84% STUDENT: You're getting a blur between your spectrum pieces 01:02:21.280 --> 01:02:22.780 align:middle line:90% exactly. 01:02:22.780 --> 01:02:25.360 align:middle line:84% PROFESSOR: The blur will be the wavelength as opposed 01:02:25.360 --> 01:02:27.040 align:middle line:90% to its space. 01:02:27.040 --> 01:02:32.810 align:middle line:84% Because even if you make this pinhold larger, 01:02:32.810 --> 01:02:38.150 align:middle line:84% conceptually, I guess you want to follow this trend. 01:02:38.150 --> 01:02:40.100 align:middle line:90% STUDENT: Well, that's-- 01:02:40.100 --> 01:02:43.040 align:middle line:84% PROFESSOR: Well, just to get your thinking, if you increase 01:02:43.040 --> 01:02:46.850 align:middle line:84% that pinhole, you will still get an image 01:02:46.850 --> 01:02:51.980 align:middle line:84% at the end that looks sharp just like [INAUDIBLE] image. 01:02:51.980 --> 01:02:54.050 align:middle line:90% Because we created two images. 01:02:54.050 --> 01:02:57.560 align:middle line:84% The prism plane is basically some kind of a virtual image. 01:02:57.560 --> 01:02:58.820 align:middle line:90% We formed an image on that. 01:02:58.820 --> 01:03:02.600 align:middle line:84% And the image of that is being formed on the sensor. 01:03:02.600 --> 01:03:07.610 align:middle line:84% So, there's no problem in terms of the focus blur. 01:03:07.610 --> 01:03:11.600 align:middle line:84% But the ability to control specific wavelengths 01:03:11.600 --> 01:03:13.160 align:middle line:90% has now decreased. 01:03:13.160 --> 01:03:16.130 align:middle line:84% So, there is a blur in terms of wavelengths, 01:03:16.130 --> 01:03:18.800 align:middle line:84% spectral copies are still missing. 01:03:18.800 --> 01:03:21.080 align:middle line:84% STUDENT: So what if we put some kind of like pattern 01:03:21.080 --> 01:03:24.227 align:middle line:84% to light in more light but then we can [INAUDIBLE] out later. 01:03:24.227 --> 01:03:26.060 align:middle line:84% PROFESSOR: Some kind of [INAUDIBLE] aperture 01:03:26.060 --> 01:03:27.018 align:middle line:90% or something like that? 01:03:27.018 --> 01:03:28.430 align:middle line:90% STUDENT: Or mask or anything. 01:03:28.430 --> 01:03:29.320 align:middle line:90% PROFESSOR: Very good. 01:03:29.320 --> 01:03:31.280 align:middle line:90% Very good Thank you. 01:03:31.280 --> 01:03:33.637 align:middle line:84% We haven't covered aperture yet so-- 01:03:33.637 --> 01:03:35.720 align:middle line:84% ANKIT MOHAN: I briefly talked about it in the end. 01:03:35.720 --> 01:03:37.160 align:middle line:90% But yeah, you're right. 01:03:37.160 --> 01:03:40.040 align:middle line:84% You could do something of that to deepen all the effects 01:03:40.040 --> 01:03:42.830 align:middle line:90% of the blur arising from this. 01:03:42.830 --> 01:03:45.950 align:middle line:84% But it turns out that this is a different kind of trade 01:03:45.950 --> 01:03:49.430 align:middle line:84% off that you from what we saw earlier like where we were 01:03:49.430 --> 01:03:50.683 align:middle line:90% doing the scanning in time. 01:03:50.683 --> 01:03:52.100 align:middle line:84% When you're scanning in time, it's 01:03:52.100 --> 01:03:54.193 align:middle line:84% almost like have a pinhole in time. 01:03:54.193 --> 01:03:55.610 align:middle line:84% And then if you have scene motion, 01:03:55.610 --> 01:03:57.890 align:middle line:84% then you would have blur due to that. 01:03:57.890 --> 01:04:00.080 align:middle line:84% In this case, you have a pinhole in space. 01:04:00.080 --> 01:04:03.800 align:middle line:84% So, you're getting all of this in one shot 01:04:03.800 --> 01:04:06.620 align:middle line:84% but you're turning of a light over there 01:04:06.620 --> 01:04:07.940 align:middle line:90% because you need this-- 01:04:07.940 --> 01:04:10.790 align:middle line:84% you can't have an infinitely-- very large aperture. 01:04:10.790 --> 01:04:14.780 align:middle line:90% And you can have some sort of-- 01:04:14.780 --> 01:04:17.960 align:middle line:84% reasonably not very tiny aperture, 01:04:17.960 --> 01:04:21.800 align:middle line:84% you can get something like six or 10 different wavelengths 01:04:21.800 --> 01:04:24.620 align:middle line:84% over here, which is not great, but it's 01:04:24.620 --> 01:04:26.420 align:middle line:84% something that gives you a control of what 01:04:26.420 --> 01:04:27.800 align:middle line:90% you can control the wavelength. 01:04:27.800 --> 01:04:30.900 align:middle line:84% The aperture size and control the fidelity of the wavelengths 01:04:30.900 --> 01:04:32.150 align:middle line:90% that you're going to get here. 01:04:32.150 --> 01:04:35.158 align:middle line:90% 01:04:35.158 --> 01:04:36.950 align:middle line:84% So, that's sort of the setup that we build. 01:04:36.950 --> 01:04:38.930 align:middle line:84% You have the image sensor over here, which 01:04:38.930 --> 01:04:41.280 align:middle line:90% is nothing but standard camera. 01:04:41.280 --> 01:04:44.690 align:middle line:84% This is the [INAUDIBLE] the mask that 01:04:44.690 --> 01:04:46.340 align:middle line:84% controls what wavelengths go through 01:04:46.340 --> 01:04:49.230 align:middle line:84% and what don't and then a bunch of lenses over here. 01:04:49.230 --> 01:04:51.660 align:middle line:84% And that's the diffraction grating that bends everything. 01:04:51.660 --> 01:04:53.240 align:middle line:84% So, one thing you would notice is 01:04:53.240 --> 01:04:56.840 align:middle line:84% that there is a bend in the optical axis, something 01:04:56.840 --> 01:04:59.030 align:middle line:84% that I didn't show in the optical diagram so far. 01:04:59.030 --> 01:05:03.050 align:middle line:84% But it's-- again, something you have to take care of when-- 01:05:03.050 --> 01:05:07.030 align:middle line:84% if you're actually building the system. 01:05:07.030 --> 01:05:10.780 align:middle line:84% So, you want me to go through all the examples or I mean? 01:05:10.780 --> 01:05:14.860 align:middle line:84% So, this is a simple test setup that we built. 01:05:14.860 --> 01:05:17.740 align:middle line:84% And the idea was we had a spectral rainbow generator. 01:05:17.740 --> 01:05:21.410 align:middle line:84% So, we have this rainbow which is going from red to blue. 01:05:21.410 --> 01:05:22.640 align:middle line:90% It's kind of like that. 01:05:22.640 --> 01:05:26.980 align:middle line:84% And then we are imaging this with our agile spectrum camera. 01:05:26.980 --> 01:05:29.080 align:middle line:84% Now, notice that the color that you see here 01:05:29.080 --> 01:05:33.530 align:middle line:84% is because of the color of the Bayer sensor On the camera. 01:05:33.530 --> 01:05:36.460 align:middle line:84% And it's really-- if you were to use a monochrome sensor, 01:05:36.460 --> 01:05:38.995 align:middle line:84% it would be all gray and just everything 01:05:38.995 --> 01:05:41.110 align:middle line:90% with equal intensity. 01:05:41.110 --> 01:05:44.390 align:middle line:84% So, first when we block off a certain wavelength, 01:05:44.390 --> 01:05:48.070 align:middle line:84% so let's say about 606 nanometers over here. 01:05:48.070 --> 01:05:50.530 align:middle line:84% You see a corresponding gap in the image 01:05:50.530 --> 01:05:52.280 align:middle line:90% that's captured by the camera. 01:05:52.280 --> 01:05:58.060 align:middle line:84% And it gives you an indication that it is blocking off 01:05:58.060 --> 01:06:01.090 align:middle line:84% in the right range and the Bayer filter actually 01:06:01.090 --> 01:06:03.200 align:middle line:90% helps you see that here. 01:06:03.200 --> 01:06:06.130 align:middle line:84% And if you put some other arbitrary mask 01:06:06.130 --> 01:06:10.330 align:middle line:84% that's actually blocking off in two regions, you get similar-- 01:06:10.330 --> 01:06:15.430 align:middle line:84% a corresponding image on the sensor. 01:06:15.430 --> 01:06:18.490 align:middle line:84% It looks very different over here. 01:06:18.490 --> 01:06:22.760 align:middle line:84% It's like more blue over here and so on. 01:06:22.760 --> 01:06:26.110 align:middle line:84% So, one of the applications you can do with this thing 01:06:26.110 --> 01:06:28.750 align:middle line:84% that we worked on was trying to reduce the layer coming out 01:06:28.750 --> 01:06:29.440 align:middle line:90% of this led. 01:06:29.440 --> 01:06:31.630 align:middle line:84% So, it's almost impossible to see here 01:06:31.630 --> 01:06:33.790 align:middle line:84% but you actually have text written 01:06:33.790 --> 01:06:38.905 align:middle line:84% in the background, which is some EG over here in the background. 01:06:38.905 --> 01:06:41.030 align:middle line:84% And then you have this right LED in the foreground. 01:06:41.030 --> 01:06:42.610 align:middle line:84% And you want to be able to capture 01:06:42.610 --> 01:06:45.490 align:middle line:84% both the background and the foreground at the same time. 01:06:45.490 --> 01:06:48.580 align:middle line:84% And the background is much, much darker than the foreground. 01:06:48.580 --> 01:06:51.910 align:middle line:84% So, if you were to do traditional high dynamic range 01:06:51.910 --> 01:06:53.830 align:middle line:84% imaging, you would-- if you increase 01:06:53.830 --> 01:06:56.440 align:middle line:84% the exposure much more, this halo that's 01:06:56.440 --> 01:06:59.020 align:middle line:84% coming out of this LED starts to occupy 01:06:59.020 --> 01:07:00.730 align:middle line:90% even more parts of the scene. 01:07:00.730 --> 01:07:03.190 align:middle line:84% And you can't really see the background. 01:07:03.190 --> 01:07:09.100 align:middle line:84% If you decrease the exposure, then this halo or this artifact 01:07:09.100 --> 01:07:09.610 align:middle line:90% reduces. 01:07:09.610 --> 01:07:11.710 align:middle line:84% But then the background becomes even darker. 01:07:11.710 --> 01:07:13.750 align:middle line:84% So, what you really want to do is just 01:07:13.750 --> 01:07:16.570 align:middle line:84% occlude this LED, the effect of the LED, 01:07:16.570 --> 01:07:18.340 align:middle line:84% and be able to see the background. 01:07:18.340 --> 01:07:21.944 align:middle line:90% And you can do that by-- 01:07:21.944 --> 01:07:23.410 align:middle line:90% you still don't see it. 01:07:23.410 --> 01:07:26.620 align:middle line:84% So, you can do that by blocking out 01:07:26.620 --> 01:07:28.600 align:middle line:84% these wavelengths corresponding to the LED. 01:07:28.600 --> 01:07:30.400 align:middle line:90% And now the LED is much dimmer. 01:07:30.400 --> 01:07:32.230 align:middle line:84% And actually, if you look at it over here, 01:07:32.230 --> 01:07:34.232 align:middle line:84% you can clearly read the background. 01:07:34.232 --> 01:07:37.140 align:middle line:90% 01:07:37.140 --> 01:07:38.395 align:middle line:90% So-- 01:07:38.395 --> 01:07:39.520 align:middle line:90% PROFESSOR: This is strange. 01:07:39.520 --> 01:07:41.230 align:middle line:90% Now, we can see [INAUDIBLE]. 01:07:41.230 --> 01:07:44.320 align:middle line:90% [INTERPOSING VOICES] 01:07:44.320 --> 01:07:47.170 align:middle line:84% ANKIT MOHAN: So, you can see the background a little bit 01:07:47.170 --> 01:07:48.620 align:middle line:90% over there but this way. 01:07:48.620 --> 01:07:52.450 align:middle line:84% So that's-- it's doing high dynamic range but by modulating 01:07:52.450 --> 01:07:56.500 align:middle line:84% the spectral profile of the scene rather than by cutting 01:07:56.500 --> 01:07:59.260 align:middle line:84% off all wavelengths equally or cutting of just certain 01:07:59.260 --> 01:08:01.593 align:middle line:84% wavelengths that you know [INAUDIBLE] the scene that are 01:08:01.593 --> 01:08:04.030 align:middle line:84% causing this disturbance in the scene. 01:08:04.030 --> 01:08:06.255 align:middle line:90% 01:08:06.255 --> 01:08:07.880 align:middle line:84% So, you can-- it turns out that you can 01:08:07.880 --> 01:08:09.770 align:middle line:90% build a similar projector also. 01:08:09.770 --> 01:08:12.320 align:middle line:84% [INAUDIBLE] was all for a camera but you can also 01:08:12.320 --> 01:08:13.680 align:middle line:90% use this as a projection system. 01:08:13.680 --> 01:08:17.060 align:middle line:84% So, now we have a traditional projector diffraction grating, 01:08:17.060 --> 01:08:20.760 align:middle line:84% this lens, and then you have this R plane, you have the mask 01:08:20.760 --> 01:08:22.310 align:middle line:90% and then the screen is up there. 01:08:22.310 --> 01:08:26.779 align:middle line:84% And the advantage of doing it with this projector system 01:08:26.779 --> 01:08:29.000 align:middle line:84% is that you already have a pinhole it turns out 01:08:29.000 --> 01:08:30.260 align:middle line:90% inside the projector. 01:08:30.260 --> 01:08:34.040 align:middle line:84% And most projectors actually have a very long optical path 01:08:34.040 --> 01:08:38.240 align:middle line:84% between the light source and the projection lens 01:08:38.240 --> 01:08:40.550 align:middle line:84% in order to increase the depth of field 01:08:40.550 --> 01:08:45.950 align:middle line:84% that they get then once they project the image. 01:08:45.950 --> 01:08:47.689 align:middle line:84% And it turns out we didn't even have 01:08:47.689 --> 01:08:50.485 align:middle line:84% to stop the lens or anything of that sort. 01:08:50.485 --> 01:08:54.120 align:middle line:84% And it just worked for a projector, 01:08:54.120 --> 01:08:56.120 align:middle line:84% a standard projector, without modifying it much. 01:08:56.120 --> 01:08:58.830 align:middle line:90% 01:08:58.830 --> 01:09:03.870 align:middle line:84% So, this is an acute example where 01:09:03.870 --> 01:09:06.149 align:middle line:84% the thing I wanted to mention over here 01:09:06.149 --> 01:09:07.500 align:middle line:90% is this concept of metamers. 01:09:07.500 --> 01:09:09.330 align:middle line:84% I don't if that's been discussed. 01:09:09.330 --> 01:09:12.300 align:middle line:84% But metamer is anything-- two colors 01:09:12.300 --> 01:09:15.990 align:middle line:84% which appear the same to the human eye or a camera 01:09:15.990 --> 01:09:18.640 align:middle line:84% when viewed under a certain type of illumination. 01:09:18.640 --> 01:09:22.109 align:middle line:84% So, this is the wide illumination of the scene 01:09:22.109 --> 01:09:25.140 align:middle line:84% and you probably can't see this, like this orange cloth 01:09:25.140 --> 01:09:26.979 align:middle line:90% or blue cloth and so on. 01:09:26.979 --> 01:09:29.069 align:middle line:84% And many of the colors over here, 01:09:29.069 --> 01:09:31.350 align:middle line:90% you can probably see there. 01:09:31.350 --> 01:09:34.064 align:middle line:84% This-- it's very hard to distinguish between the start 01:09:34.064 --> 01:09:34.939 align:middle line:90% and the [INAUDIBLE]-- 01:09:34.939 --> 01:09:35.772 align:middle line:90% [INTERPOSING VOICES] 01:09:35.772 --> 01:09:37.550 align:middle line:84% PROFESSOR: Is that 3M stickies, right? 01:09:37.550 --> 01:09:37.740 align:middle line:90% ANKIT MOHAN: Yeah. 01:09:37.740 --> 01:09:38.490 align:middle line:90% These are all stickies. 01:09:38.490 --> 01:09:39.948 align:middle line:84% So, these are actually fluorescent, 01:09:39.948 --> 01:09:41.859 align:middle line:90% these envelopes I think. 01:09:41.859 --> 01:09:44.069 align:middle line:90% And if you look on the-- 01:09:44.069 --> 01:09:48.439 align:middle line:90% 01:09:48.439 --> 01:09:49.819 align:middle line:84% so, if you look over here, you'll 01:09:49.819 --> 01:09:54.470 align:middle line:84% see as we project different wavelengths, how these colors 01:09:54.470 --> 01:09:57.140 align:middle line:84% they appear almost black and very, very similar to one 01:09:57.140 --> 01:09:58.760 align:middle line:84% another under different illumination. 01:09:58.760 --> 01:10:02.400 align:middle line:84% And now, one of them is going to become brighter than the other. 01:10:02.400 --> 01:10:04.410 align:middle line:84% And you can actually see the difference. 01:10:04.410 --> 01:10:12.410 align:middle line:84% So, it's very interesting to see the progression of how-- 01:10:12.410 --> 01:10:13.918 align:middle line:84% it gives you an intuition into that 01:10:13.918 --> 01:10:15.710 align:middle line:84% there's much more to the wavelength profile 01:10:15.710 --> 01:10:18.420 align:middle line:84% than what the human eye simply sees in white light. 01:10:18.420 --> 01:10:20.330 align:middle line:84% STUDENT: So, can you just explain again 01:10:20.330 --> 01:10:22.348 align:middle line:84% what happened I mean, during this animation? 01:10:22.348 --> 01:10:23.390 align:middle line:90% ANKIT MOHAN: Well, this-- 01:10:23.390 --> 01:10:27.410 align:middle line:84% I'm just projecting monochromatic very narrow 01:10:27.410 --> 01:10:28.430 align:middle line:90% wavelength of light. 01:10:28.430 --> 01:10:30.290 align:middle line:84% PROFESSOR: [INAUDIBLE] projecting red light. 01:10:30.290 --> 01:10:31.207 align:middle line:90% ANKIT MOHAN: Yes, Yes. 01:10:31.207 --> 01:10:32.610 align:middle line:90% So red, green, and blue. 01:10:32.610 --> 01:10:36.450 align:middle line:84% So, about 12 or 15 different wavelengths. 01:10:36.450 --> 01:10:38.840 align:middle line:84% PROFESSOR: So, it starts with very reddish, slightly 01:10:38.840 --> 01:10:43.520 align:middle line:84% yellowish, then greenish, then magenta, and then finally, 01:10:43.520 --> 01:10:45.320 align:middle line:90% bluish-- 01:10:45.320 --> 01:10:47.690 align:middle line:84% not magenta, cyan and then bluish. 01:10:47.690 --> 01:10:50.030 align:middle line:84% ANKIT MOHAN: It just shows you some of the-- 01:10:50.030 --> 01:10:52.070 align:middle line:84% I mean, I think if you look at the background, 01:10:52.070 --> 01:10:56.870 align:middle line:84% does this looks so different from this envelope 01:10:56.870 --> 01:10:58.880 align:middle line:84% whereas over here, it's actually very similar 01:10:58.880 --> 01:11:02.600 align:middle line:90% color in white light. 01:11:02.600 --> 01:11:09.500 align:middle line:84% PROFESSOR: And these papers are very fluorescent intentionally, 01:11:09.500 --> 01:11:12.920 align:middle line:84% so they look very beautiful when they're on your tabletop. 01:11:12.920 --> 01:11:15.530 align:middle line:84% But they're also responding to very narrow wavelengths 01:11:15.530 --> 01:11:17.490 align:middle line:90% of light. 01:11:17.490 --> 01:11:22.610 align:middle line:84% So, that's why some of these envelopes 01:11:22.610 --> 01:11:25.010 align:middle line:84% become completely dark when you eliminate 01:11:25.010 --> 01:11:28.074 align:middle line:90% all your particular wavelength. 01:11:28.074 --> 01:11:30.120 align:middle line:90% STUDENT: I have a question. 01:11:30.120 --> 01:11:32.790 align:middle line:84% Could it have been possible for you to just take an image 01:11:32.790 --> 01:11:36.510 align:middle line:84% and then use different channels to gender those 15 01:11:36.510 --> 01:11:38.010 align:middle line:90% images that you generated? 01:11:38.010 --> 01:11:41.150 align:middle line:84% Did you have to definitely take them at different illuminations 01:11:41.150 --> 01:11:41.950 align:middle line:90% or-- 01:11:41.950 --> 01:11:44.580 align:middle line:84% ANKIT MOHAN: Or if you have a camera that captures all 15-- 01:11:44.580 --> 01:11:45.977 align:middle line:90% [INTERPOSING VOICES] 01:11:45.977 --> 01:11:47.310 align:middle line:90% MICHAEL: Broadband illumination. 01:11:47.310 --> 01:11:48.210 align:middle line:84% Those are the two things you need, 01:11:48.210 --> 01:11:50.335 align:middle line:84% broadband illumination and a camera that captures-- 01:11:50.335 --> 01:11:51.480 align:middle line:90% [INTERPOSING VOICES] 01:11:51.480 --> 01:11:53.435 align:middle line:90% PROFESSOR: 15 spectrums. 01:11:53.435 --> 01:11:54.810 align:middle line:84% If you take only three spectrums, 01:11:54.810 --> 01:11:57.340 align:middle line:84% then you only have three numbers of pixels. 01:11:57.340 --> 01:11:59.682 align:middle line:84% So, there is no way you can recover the 15. 01:11:59.682 --> 01:12:01.140 align:middle line:84% And if you have a camera like this, 01:12:01.140 --> 01:12:03.690 align:middle line:84% then [INAUDIBLE] other applications like 01:12:03.690 --> 01:12:08.070 align:middle line:84% distinguishing between a fake vegetable and real vegetable 01:12:08.070 --> 01:12:11.140 align:middle line:84% and freshness of skin and looking through fog-- 01:12:11.140 --> 01:12:11.640 align:middle line:90% all those-- 01:12:11.640 --> 01:12:15.900 align:middle line:84% not fog maybe, but some of the problems become manageable. 01:12:15.900 --> 01:12:19.710 align:middle line:84% And film and even digital photography 01:12:19.710 --> 01:12:22.180 align:middle line:84% is just trying to mimic human eye, three colors, 01:12:22.180 --> 01:12:24.858 align:middle line:84% in fact, three fixed colors as Ankit was saying. 01:12:24.858 --> 01:12:26.400 align:middle line:84% But what you really want is something 01:12:26.400 --> 01:12:27.930 align:middle line:90% like [INAUDIBLE] equalizer. 01:12:27.930 --> 01:12:31.240 align:middle line:84% You can tune any frequency, any wavelength 01:12:31.240 --> 01:12:33.300 align:middle line:84% so you can see the world in interesting ways. 01:12:33.300 --> 01:12:36.930 align:middle line:90% 01:12:36.930 --> 01:12:42.470 align:middle line:84% ANKIT MOHAN: So, this is another example of where adaptive color 01:12:42.470 --> 01:12:43.640 align:middle line:90% parameters would help. 01:12:43.640 --> 01:12:47.450 align:middle line:84% And as I mentioned earlier, there is very little-- 01:12:47.450 --> 01:12:48.860 align:middle line:84% cyan is one of the hardest colors 01:12:48.860 --> 01:12:51.740 align:middle line:84% to predict for most projectors because your color 01:12:51.740 --> 01:12:53.300 align:middle line:90% gamut is usually over there. 01:12:53.300 --> 01:12:55.520 align:middle line:84% And there's very little representation from there. 01:12:55.520 --> 01:12:57.830 align:middle line:84% And it turns out there are two definitions of cyan. 01:12:57.830 --> 01:13:00.170 align:middle line:84% One is the traditional printing cyan 01:13:00.170 --> 01:13:02.940 align:middle line:84% and the other is what's called the electronic or electric 01:13:02.940 --> 01:13:03.440 align:middle line:90% cyan. 01:13:03.440 --> 01:13:07.070 align:middle line:84% And this color is-- so what I'm projecting here 01:13:07.070 --> 01:13:09.440 align:middle line:84% is just a ramp between blue and green. 01:13:09.440 --> 01:13:12.030 align:middle line:84% So, this is blue here and green and colors in between. 01:13:12.030 --> 01:13:14.690 align:middle line:84% And this is what the computer thinks of cyan 01:13:14.690 --> 01:13:15.680 align:middle line:90% at the top and bottom. 01:13:15.680 --> 01:13:17.940 align:middle line:84% And you can see there's clearly a leak over here. 01:13:17.940 --> 01:13:21.440 align:middle line:90% And that's because when you-- 01:13:21.440 --> 01:13:24.110 align:middle line:84% along this line is what I'm projecting over here, 01:13:24.110 --> 01:13:25.670 align:middle line:90% this line. 01:13:25.670 --> 01:13:29.150 align:middle line:84% And so this cyan is nothing but a point that lies somewhere 01:13:29.150 --> 01:13:30.180 align:middle line:90% on this line. 01:13:30.180 --> 01:13:33.890 align:middle line:84% So, you cannot project a color which is out here using 01:13:33.890 --> 01:13:36.480 align:middle line:84% a standard projector or a display. 01:13:36.480 --> 01:13:39.740 align:middle line:84% But if you use this [INAUDIBLE] spectrum, 01:13:39.740 --> 01:13:43.040 align:middle line:84% you can tweak it so that your color over here 01:13:43.040 --> 01:13:48.770 align:middle line:84% is actually something that's outside your color spectrum 01:13:48.770 --> 01:13:50.583 align:middle line:84% that the projector could have displayed. 01:13:50.583 --> 01:13:52.250 align:middle line:84% That's because you're using-- you're not 01:13:52.250 --> 01:13:54.320 align:middle line:84% using the filters that the projector was 01:13:54.320 --> 01:13:56.757 align:middle line:84% using but actually displaying something outside it. 01:13:56.757 --> 01:13:58.340 align:middle line:84% PROFESSOR: So, some companies are also 01:13:58.340 --> 01:14:01.910 align:middle line:84% trying to sell you four or six color projectors. 01:14:01.910 --> 01:14:06.020 align:middle line:84% So, they'll have this point here, maybe another point here, 01:14:06.020 --> 01:14:08.520 align:middle line:84% another point here, here, here, and here 01:14:08.520 --> 01:14:11.600 align:middle line:84% so they can cover more colors than a standard three color 01:14:11.600 --> 01:14:12.200 align:middle line:90% projector. 01:14:12.200 --> 01:14:14.210 align:middle line:84% Because there is a three color projector, 01:14:14.210 --> 01:14:16.820 align:middle line:90% you must pick some three points. 01:14:16.820 --> 01:14:20.480 align:middle line:84% And you cannot represent this shape with three points. 01:14:20.480 --> 01:14:23.570 align:middle line:84% If you start going to close to over here, if you take 01:14:23.570 --> 01:14:26.540 align:middle line:84% this part too close here, then as Ankit said earlier, 01:14:26.540 --> 01:14:28.527 align:middle line:90% it has to be very pure green. 01:14:28.527 --> 01:14:30.860 align:middle line:84% And that's difficult to generate unless you have a laser 01:14:30.860 --> 01:14:32.258 align:middle line:90% projection. 01:14:32.258 --> 01:14:34.550 align:middle line:84% STUDENT: So, do the laser projectors have a wider color 01:14:34.550 --> 01:14:34.730 align:middle line:90% gamut? 01:14:34.730 --> 01:14:35.180 align:middle line:90% PROFESSOR: Exactly. 01:14:35.180 --> 01:14:37.220 align:middle line:84% So, the laser projections can go off-- they 01:14:37.220 --> 01:14:41.720 align:middle line:84% can stay almost on the pure colors. 01:14:41.720 --> 01:14:44.850 align:middle line:84% So they can go all the way here and here and here. 01:14:44.850 --> 01:14:47.180 align:middle line:84% So, they can cover a larger gamut. 01:14:47.180 --> 01:14:49.062 align:middle line:84% But there are other issues with that. 01:14:49.062 --> 01:14:51.270 align:middle line:84% STUDENT: So, they usually end up with higher contrast 01:14:51.270 --> 01:14:54.410 align:middle line:84% but they are less brightness or-- 01:14:54.410 --> 01:14:55.627 align:middle line:90% how do they usually? 01:14:55.627 --> 01:14:57.710 align:middle line:84% PROFESSOR: They have to have good color rendition. 01:14:57.710 --> 01:14:59.400 align:middle line:90% STUDENT: Yeah. 01:14:59.400 --> 01:15:02.510 align:middle line:84% PROFESSOR: [INAUDIBLE] a problem. 01:15:02.510 --> 01:15:04.700 align:middle line:90% [INTERPOSING VOICES] 01:15:04.700 --> 01:15:07.640 align:middle line:84% Yeah, so in general, a laser projector 01:15:07.640 --> 01:15:10.670 align:middle line:84% is going to create a very nice rendition. 01:15:10.670 --> 01:15:13.100 align:middle line:84% It's just that it's not compatible directly 01:15:13.100 --> 01:15:15.630 align:middle line:90% with the human vision. 01:15:15.630 --> 01:15:19.310 align:middle line:84% So, sensitivity about the human eye for pure color 01:15:19.310 --> 01:15:21.868 align:middle line:90% is not so great, unfortunately. 01:15:21.868 --> 01:15:24.410 align:middle line:84% ANKIT MOHAN: And there Are some constraints in the wavelength 01:15:24.410 --> 01:15:25.140 align:middle line:90% that you can use. 01:15:25.140 --> 01:15:26.510 align:middle line:84% You can't just choose any wavelength. 01:15:26.510 --> 01:15:28.260 align:middle line:84% They have to have a very specific problem. 01:15:28.260 --> 01:15:30.710 align:middle line:84% Then they use frequency doublers or something 01:15:30.710 --> 01:15:32.510 align:middle line:90% to get the other colors. 01:15:32.510 --> 01:15:35.520 align:middle line:84% It's just a little more cumbersome to do that. 01:15:35.520 --> 01:15:37.910 align:middle line:84% And then there's the issue of power 01:15:37.910 --> 01:15:39.710 align:middle line:84% that you need if you want the laser 01:15:39.710 --> 01:15:41.180 align:middle line:90% projector to be bright enough. 01:15:41.180 --> 01:15:44.790 align:middle line:84% You need to have very bright lasers. 01:15:44.790 --> 01:15:46.820 align:middle line:90% MICHAEL: Yeah. 01:15:46.820 --> 01:15:50.690 align:middle line:84% The amount of optical power that a projector throws out 01:15:50.690 --> 01:15:53.460 align:middle line:84% compared to a typical laser is quite large. 01:15:53.460 --> 01:15:56.090 align:middle line:84% So these things actually put out a lot of power 01:15:56.090 --> 01:15:59.040 align:middle line:84% compared to your laser pointer for example. 01:15:59.040 --> 01:16:01.130 align:middle line:84% So, putting a laser bright enough 01:16:01.130 --> 01:16:04.310 align:middle line:90% to generate this much light-- 01:16:04.310 --> 01:16:06.000 align:middle line:90% yeah, that is challenging. 01:16:06.000 --> 01:16:06.980 align:middle line:90% ANKIT MOHAN: Yeah. 01:16:06.980 --> 01:16:09.980 align:middle line:84% Yeah, so-- I mean, this looks bright 01:16:09.980 --> 01:16:12.680 align:middle line:84% because it's illuminating just one point, the laser pointer. 01:16:12.680 --> 01:16:14.510 align:middle line:84% But if you were to actually distribute 01:16:14.510 --> 01:16:16.843 align:middle line:84% this over the whole region, you would barely see it 01:16:16.843 --> 01:16:18.260 align:middle line:84% STUDENT: Most of them are scanning 01:16:18.260 --> 01:16:19.968 align:middle line:90% a single point, aren't they? 01:16:19.968 --> 01:16:21.260 align:middle line:90% Aren't most laser projections-- 01:16:21.260 --> 01:16:22.460 align:middle line:90% PROFESSOR: Yes, exactly. 01:16:22.460 --> 01:16:23.240 align:middle line:90% ANKIT MOHAN: Yeah. 01:16:23.240 --> 01:16:26.237 align:middle line:84% So, it's as though you're actually distributing 01:16:26.237 --> 01:16:27.320 align:middle line:90% this over the whole space. 01:16:27.320 --> 01:16:30.740 align:middle line:84% PROFESSOR: So if you have a million pixels, all one frame, 01:16:30.740 --> 01:16:34.615 align:middle line:84% every pixel is illuminated only one millionth of a frame. 01:16:34.615 --> 01:16:35.990 align:middle line:84% MICHAEL: That's the trade, right? 01:16:35.990 --> 01:16:39.560 align:middle line:84% Is you still need a lot of power because you're not spending 01:16:39.560 --> 01:16:41.310 align:middle line:90% much time in any one place. 01:16:41.310 --> 01:16:44.220 align:middle line:90% PROFESSOR: [INAUDIBLE] problem? 01:16:44.220 --> 01:16:48.795 align:middle line:84% I'm very, very hopeful that solid-state lasers will-- 01:16:48.795 --> 01:16:50.420 align:middle line:84% MICHAEL: Actually, that's [INAUDIBLE].. 01:16:50.420 --> 01:16:52.880 align:middle line:84% If we assume for the moment that laser pointer is 01:16:52.880 --> 01:16:58.790 align:middle line:84% as bright as the red chunk of that slide you were showing us, 01:16:58.790 --> 01:17:01.790 align:middle line:84% like on the previous one, if it's roughly 01:17:01.790 --> 01:17:06.050 align:middle line:84% is bright, same ballpark, but that's 01:17:06.050 --> 01:17:08.335 align:middle line:90% roughly a megapixel image-- 01:17:08.335 --> 01:17:09.710 align:middle line:84% PROFESSOR: A million times more-- 01:17:09.710 --> 01:17:11.460 align:middle line:84% MICHAEL: About a million times more power. 01:17:11.460 --> 01:17:13.028 align:middle line:90% ANKIT MOHAN: Yeah. 01:17:13.028 --> 01:17:14.070 align:middle line:90% PROFESSOR: Which is true. 01:17:14.070 --> 01:17:15.950 align:middle line:90% This is about one milliwat. 01:17:15.950 --> 01:17:17.610 align:middle line:90% And the project is 250 watt. 01:17:17.610 --> 01:17:20.170 align:middle line:90% 01:17:20.170 --> 01:17:23.770 align:middle line:84% It's 250 million times more the [INAUDIBLE] 01:17:23.770 --> 01:17:26.610 align:middle line:90% compared to the [INAUDIBLE]. 01:17:26.610 --> 01:17:28.610 align:middle line:84% ANKIT MOHAN: I think laser projectors work great 01:17:28.610 --> 01:17:31.280 align:middle line:84% if you are in this room where it's not too bright 01:17:31.280 --> 01:17:33.720 align:middle line:84% and you had the projector here [INAUDIBLE].. 01:17:33.720 --> 01:17:35.850 align:middle line:84% It works reasonably well, but not 01:17:35.850 --> 01:17:40.020 align:middle line:84% if you have the lights on you probably won't see anything. 01:17:40.020 --> 01:17:42.930 align:middle line:84% So, another thing you can do with this is don't-- 01:17:42.930 --> 01:17:45.030 align:middle line:84% since you're not stuck with those three color 01:17:45.030 --> 01:17:46.228 align:middle line:90% primaries anymore. 01:17:46.228 --> 01:17:47.520 align:middle line:90% So, this is just a scene again. 01:17:47.520 --> 01:17:48.900 align:middle line:84% It doesn't show up there as well. 01:17:48.900 --> 01:17:51.670 align:middle line:84% You have blue over here and you have yellow over here. 01:17:51.670 --> 01:17:55.980 align:middle line:84% And so traditionally, you would have a RGB filter 01:17:55.980 --> 01:18:00.150 align:middle line:84% where for yellow, you're turning it on in the red 01:18:00.150 --> 01:18:05.190 align:middle line:84% and the green color filter is in front of your projection. 01:18:05.190 --> 01:18:06.840 align:middle line:84% And for the blue, you are turning it 01:18:06.840 --> 01:18:11.700 align:middle line:84% on only when the blue part of the color is in front. 01:18:11.700 --> 01:18:15.030 align:middle line:84% But if you know that your scene has just as yellow and blue, 01:18:15.030 --> 01:18:17.290 align:middle line:84% you can actually just project a yellow. 01:18:17.290 --> 01:18:20.310 align:middle line:84% And you can use a yellow and blue projector. 01:18:20.310 --> 01:18:23.220 align:middle line:84% And you don't have to use the same traditional fixed color 01:18:23.220 --> 01:18:23.940 align:middle line:90% primaries. 01:18:23.940 --> 01:18:26.610 align:middle line:84% And this will give you a brighter scene and one 01:18:26.610 --> 01:18:28.500 align:middle line:90% that has more saturation also. 01:18:28.500 --> 01:18:32.170 align:middle line:90% 01:18:32.170 --> 01:18:35.290 align:middle line:84% So, one other cube example is that 01:18:35.290 --> 01:18:39.340 align:middle line:84% of colorblindness simulation [INAUDIBLE] 01:18:39.340 --> 01:18:43.030 align:middle line:84% Professor Emanuel has done a lot of work also in this area, 01:18:43.030 --> 01:18:43.630 align:middle line:90% I think. 01:18:43.630 --> 01:18:48.310 align:middle line:84% And the idea over here is so one of the most popular types 01:18:48.310 --> 01:18:50.343 align:middle line:84% of colorblindness is this different-- 01:18:50.343 --> 01:18:52.510 align:middle line:84% you can't tell the difference between red and green. 01:18:52.510 --> 01:18:54.130 align:middle line:84% And I think it's called [INAUDIBLE].. 01:18:54.130 --> 01:18:58.510 align:middle line:84% And so when you have white light illumination, you can-- 01:18:58.510 --> 01:19:04.600 align:middle line:84% I think, on the right and it's-- so, the lower ones are due 01:19:04.600 --> 01:19:07.540 align:middle line:84% to [INAUDIBLE] so you cannot tell the difference between 01:19:07.540 --> 01:19:10.720 align:middle line:84% the rose and the leaves and they both appear at the same color. 01:19:10.720 --> 01:19:13.220 align:middle line:84% Again, it's much easier to see over here. 01:19:13.220 --> 01:19:16.090 align:middle line:84% But if you actually project a certain wavelength 01:19:16.090 --> 01:19:19.480 align:middle line:84% of light on the object, it becomes much easier 01:19:19.480 --> 01:19:21.520 align:middle line:84% to tell the difference between these two. 01:19:21.520 --> 01:19:23.597 align:middle line:84% And it-- you can again, clearly see on the-- 01:19:23.597 --> 01:19:24.430 align:middle line:90% [INTERPOSING VOICES] 01:19:24.430 --> 01:19:25.450 align:middle line:90% PROFESSOR: [INAUDIBLE] 01:19:25.450 --> 01:19:26.290 align:middle line:90% ANKIT MOHAN: Yeah. 01:19:26.290 --> 01:19:28.520 align:middle line:84% You can actually see the difference between those two 01:19:28.520 --> 01:19:31.720 align:middle line:84% when you have the certain projection. 01:19:31.720 --> 01:19:35.170 align:middle line:84% So conceivably, using these kind of projections 01:19:35.170 --> 01:19:39.590 align:middle line:84% or using these kind of filters, you can help solve the problem. 01:19:39.590 --> 01:19:43.750 align:middle line:84% You can at least get some help in telling 01:19:43.750 --> 01:19:46.990 align:middle line:84% the difference between colors for people with colorblindness. 01:19:46.990 --> 01:19:50.180 align:middle line:90% 01:19:50.180 --> 01:19:53.280 align:middle line:84% So, [INAUDIBLE] go over some of the limitations. 01:19:53.280 --> 01:19:54.530 align:middle line:90% There's diffraction artifacts. 01:19:54.530 --> 01:19:56.420 align:middle line:84% Because you have a pinhole, you need 01:19:56.420 --> 01:19:58.100 align:middle line:84% a reasonably small f number in order 01:19:58.100 --> 01:20:03.350 align:middle line:84% to get a large number or even some different color bands. 01:20:03.350 --> 01:20:08.600 align:middle line:84% And that's basically the limitations. 01:20:08.600 --> 01:20:13.460 align:middle line:84% And so future work, one of which you just mentioned 01:20:13.460 --> 01:20:15.170 align:middle line:84% is actually using a mask over here. 01:20:15.170 --> 01:20:17.250 align:middle line:84% And I'll go into that just a little bit. 01:20:17.250 --> 01:20:21.098 align:middle line:84% And one of interesting cubed application, 01:20:21.098 --> 01:20:22.640 align:middle line:84% there's actually a company which does 01:20:22.640 --> 01:20:27.680 align:middle line:84% that they use this RGB wavelength multiplexing 01:20:27.680 --> 01:20:29.810 align:middle line:90% in order to do red and light-- 01:20:29.810 --> 01:20:32.420 align:middle line:84% the red and right eye separation. 01:20:32.420 --> 01:20:33.845 align:middle line:90% So, they project using-- 01:20:33.845 --> 01:20:35.300 align:middle line:90% PROFESSOR: [INAUDIBLE] 01:20:35.300 --> 01:20:37.400 align:middle line:90% ANKIT MOHAN: Left and right. 01:20:37.400 --> 01:20:38.750 align:middle line:90% Left and right. 01:20:38.750 --> 01:20:42.980 align:middle line:84% And so they use these projectors which have-- 01:20:42.980 --> 01:20:45.080 align:middle line:84% when you view them with naked eye, 01:20:45.080 --> 01:20:47.573 align:middle line:84% both this and this appear blue because they're 01:20:47.573 --> 01:20:48.740 align:middle line:90% both in the blue wavelength. 01:20:48.740 --> 01:20:50.930 align:middle line:84% But they're actually different parts 01:20:50.930 --> 01:20:53.930 align:middle line:84% of the wavelength that's being occupied by those two 01:20:53.930 --> 01:20:54.630 align:middle line:90% projections. 01:20:54.630 --> 01:20:58.010 align:middle line:84% And if you put a color filter in front of your eyes, a different 01:20:58.010 --> 01:21:00.650 align:middle line:84% for each eye, you can have the right eye only 01:21:00.650 --> 01:21:02.910 align:middle line:84% see this part and the right eye see with that part. 01:21:02.910 --> 01:21:05.210 align:middle line:84% So, when you view it visually, it 01:21:05.210 --> 01:21:07.670 align:middle line:84% would appear just like a normal projection. 01:21:07.670 --> 01:21:10.037 align:middle line:84% But if you put on the eyeglasses, 01:21:10.037 --> 01:21:11.870 align:middle line:84% you can actually distinguish between the two 01:21:11.870 --> 01:21:17.930 align:middle line:84% and you have multiplexing in wavelength rather than in time 01:21:17.930 --> 01:21:19.580 align:middle line:90% or in polarization. 01:21:19.580 --> 01:21:24.390 align:middle line:90% 01:21:24.390 --> 01:21:26.400 align:middle line:84% So far, we've talked about what's 01:21:26.400 --> 01:21:27.780 align:middle line:90% called the spectral light field. 01:21:27.780 --> 01:21:29.197 align:middle line:84% But it turns out, you can actually 01:21:29.197 --> 01:21:32.730 align:middle line:84% combine the concept of a light field with the spectral light. 01:21:32.730 --> 01:21:35.160 align:middle line:84% So, what we have here is a traditional [INAUDIBLE],, 01:21:35.160 --> 01:21:36.480 align:middle line:90% now it's x and theta. 01:21:36.480 --> 01:21:39.180 align:middle line:84% And you've placed a prism in front of this. 01:21:39.180 --> 01:21:43.260 align:middle line:84% So, what this prism has done is in a direction that 01:21:43.260 --> 01:21:46.290 align:middle line:84% depends on the distance between the x plane and the prism 01:21:46.290 --> 01:21:49.830 align:middle line:84% plane, you're actually going out and smearing it 01:21:49.830 --> 01:21:52.030 align:middle line:84% and getting all the wavelengths in that direction. 01:21:52.030 --> 01:21:56.460 align:middle line:84% So, it gives you what looks kind of like that 01:21:56.460 --> 01:21:57.580 align:middle line:90% or something like that. 01:21:57.580 --> 01:22:00.150 align:middle line:84% So, you have a whole bunch of overlapping spectral light 01:22:00.150 --> 01:22:02.360 align:middle line:84% fields on a spatial light field or you 01:22:02.360 --> 01:22:03.610 align:middle line:90% can think of it the other way. 01:22:03.610 --> 01:22:07.020 align:middle line:84% You have a whole bunch of overlapping spatial light 01:22:07.020 --> 01:22:08.650 align:middle line:84% fields of a spectral light field. 01:22:08.650 --> 01:22:12.510 align:middle line:84% So, I mean, you can either put x theta here and lambda here 01:22:12.510 --> 01:22:14.940 align:middle line:84% or you can replace it and put lambda here and theta here. 01:22:14.940 --> 01:22:17.140 align:middle line:84% And it's really not very different. 01:22:17.140 --> 01:22:19.530 align:middle line:84% But what becomes interesting is the case 01:22:19.530 --> 01:22:22.650 align:middle line:84% where this plane is diffused in front of parallel. 01:22:22.650 --> 01:22:24.600 align:middle line:84% In which case, each of these rays 01:22:24.600 --> 01:22:27.090 align:middle line:84% are exactly the same intensity because it's 01:22:27.090 --> 01:22:29.050 align:middle line:90% a [INAUDIBLE] or diffuse scene. 01:22:29.050 --> 01:22:33.240 align:middle line:90% So in this case, what you get-- 01:22:33.240 --> 01:22:34.890 align:middle line:84% the light field that you get over here 01:22:34.890 --> 01:22:38.820 align:middle line:84% is the special light field that we call a blurred light field. 01:22:38.820 --> 01:22:40.150 align:middle line:90% It's because you have-- 01:22:40.150 --> 01:22:42.120 align:middle line:84% you're taking the standard spectral light 01:22:42.120 --> 01:22:44.620 align:middle line:84% and then blurring it in this direction. 01:22:44.620 --> 01:22:50.640 align:middle line:84% So, what you could do is what Kevin was pointing out is use-- 01:22:50.640 --> 01:22:53.430 align:middle line:84% if this blurring is not just a box function, 01:22:53.430 --> 01:22:56.580 align:middle line:84% but a special coded function, you could then invert that 01:22:56.580 --> 01:22:59.340 align:middle line:84% and you could get the full spectral light field out of it 01:22:59.340 --> 01:23:01.230 align:middle line:90% without having to use a pinhole. 01:23:01.230 --> 01:23:07.360 align:middle line:84% So, this is what that design looks like. 01:23:07.360 --> 01:23:09.300 align:middle line:90% So you can put-- 01:23:09.300 --> 01:23:13.080 align:middle line:84% so, before I get into that, one of the things I didn't talk 01:23:13.080 --> 01:23:15.428 align:middle line:84% about is so you can-- by putting a pinhole 01:23:15.428 --> 01:23:17.220 align:middle line:84% and putting a light field camera over here, 01:23:17.220 --> 01:23:20.160 align:middle line:84% you can simply by using a standard lightweight camera, 01:23:20.160 --> 01:23:22.090 align:middle line:84% you can capture a multi-spectral image. 01:23:22.090 --> 01:23:24.180 align:middle line:84% So you can build a multi-spectral image 01:23:24.180 --> 01:23:27.000 align:middle line:84% by taking a light field camera and putting a prism, a lens, 01:23:27.000 --> 01:23:28.330 align:middle line:90% and a pinhole in front of it. 01:23:28.330 --> 01:23:31.920 align:middle line:84% So, what you're essentially doing here is replacing the-- 01:23:31.920 --> 01:23:34.447 align:middle line:84% is getting rid of the theta component of the light field 01:23:34.447 --> 01:23:37.030 align:middle line:84% and replacing it with the lambda component of the light field. 01:23:37.030 --> 01:23:39.143 align:middle line:84% So, once again, you had this light field 01:23:39.143 --> 01:23:41.310 align:middle line:84% and now you put a light field camera in front of it, 01:23:41.310 --> 01:23:44.400 align:middle line:90% you can sense that data. 01:23:44.400 --> 01:23:49.855 align:middle line:84% And this is initial results that we had in that where each-- 01:23:49.855 --> 01:23:52.230 align:middle line:84% it's basically a microlens [INAUDIBLE] that's placed next 01:23:52.230 --> 01:23:55.067 align:middle line:84% to the sensor and so you can see-- 01:23:55.067 --> 01:23:56.400 align:middle line:90% you still have the Bayer mosaic. 01:23:56.400 --> 01:23:59.220 align:middle line:84% So, you have this red to blue back with green 01:23:59.220 --> 01:24:02.520 align:middle line:84% in between, which is what it's sensing. 01:24:02.520 --> 01:24:04.440 align:middle line:84% But another thing you can do is put 01:24:04.440 --> 01:24:07.410 align:middle line:84% what's called a linearly variable interference filter. 01:24:07.410 --> 01:24:09.510 align:middle line:84% So, this filter allows only red wavelengths 01:24:09.510 --> 01:24:12.120 align:middle line:84% to come through at one end green in the middle 01:24:12.120 --> 01:24:13.480 align:middle line:90% and blue at the other end. 01:24:13.480 --> 01:24:15.390 align:middle line:84% And this is a very spectrally pure filter. 01:24:15.390 --> 01:24:20.790 align:middle line:84% It's basically built out of multiple layers of coating 01:24:20.790 --> 01:24:22.260 align:middle line:90% and using interference. 01:24:22.260 --> 01:24:26.790 align:middle line:84% And what this does is that it replaces the theta component 01:24:26.790 --> 01:24:29.070 align:middle line:84% of the light field with the lambda component 01:24:29.070 --> 01:24:30.828 align:middle line:90% of the spectral light field. 01:24:30.828 --> 01:24:33.120 align:middle line:84% And now if you put a light field camera in front of it, 01:24:33.120 --> 01:24:35.970 align:middle line:84% you can again capture the complete light 01:24:35.970 --> 01:24:38.340 align:middle line:84% the complete multi-spectral image. 01:24:38.340 --> 01:24:41.850 align:middle line:84% The trade over here is that any one point on this filter 01:24:41.850 --> 01:24:45.700 align:middle line:84% allows only one very narrow wavelength to go through. 01:24:45.700 --> 01:24:48.630 align:middle line:84% So, it's replacing a pinhole in space 01:24:48.630 --> 01:24:51.070 align:middle line:90% with a pinhole in wavelength. 01:24:51.070 --> 01:24:55.170 align:middle line:84% And so again, you're going to get very little light. 01:24:55.170 --> 01:24:58.200 align:middle line:84% But what you can also do is place instead of that filter, 01:24:58.200 --> 01:25:02.850 align:middle line:84% you can place one of these carefully designed masks, which 01:25:02.850 --> 01:25:05.970 align:middle line:84% are similar to the mask we use in the [INAUDIBLE] approach, 01:25:05.970 --> 01:25:07.920 align:middle line:84% but I guess we haven't talked about that. 01:25:07.920 --> 01:25:09.003 align:middle line:90% PROFESSOR: No, we haven't. 01:25:09.003 --> 01:25:10.602 align:middle line:84% ANKIT MOHAN: So, you can do this thing 01:25:10.602 --> 01:25:12.060 align:middle line:84% and then use deconvolution in order 01:25:12.060 --> 01:25:13.740 align:middle line:84% to recover the multispectral image. 01:25:13.740 --> 01:25:17.160 align:middle line:90% 01:25:17.160 --> 01:25:19.600 align:middle line:84% PROFESSOR: Let's take a short break and then we can-- 01:25:19.600 --> 01:25:21.600 align:middle line:84% ANKIT MOHAN: I just have this thermography stuff 01:25:21.600 --> 01:25:22.230 align:middle line:90% that you said-- 01:25:22.230 --> 01:25:23.370 align:middle line:84% PROFESSOR: Yeah, we'll do it after the break. 01:25:23.370 --> 01:25:24.287 align:middle line:90% ANKIT MOHAN: OK, cool. 01:25:24.287 --> 01:25:26.450 align:middle line:90% PROFESSOR: Cool, excellent. 01:25:26.450 --> 01:25:27.000 align:middle line:90%