Course Meeting Times
Lectures: 1 session / week, 3 hours / session
Course Description
A computational camera attempts to digitally capture the essence of visual information by exploiting the synergistic combination of task-specific optics, illumination, sensors and processing. We will discuss and play with thermal cameras, multi-spectral cameras, high-speed, and 3D range-sensing cameras and camera arrays. We will learn about opportunities in scientific and medical imaging, mobile-phone based photography, cameras for human-computer interaction (HCI) and sensors mimicking animal eyes.
We will learn about the complete camera pipeline. In several hands-on projects we will build several physical imaging prototypes and understand how each stage of the imaging process can be manipulated.
We will learn about modern methods for capturing and sharing visual information. If novel cameras can be designed to sample light in radically new ways, then rich and useful forms of visual information may be recorded — beyond those present in traditional photographs. Furthermore, if computational process can be made aware of these novel imaging models, them the scene can be analyzed in higher dimensions and novel aesthetic renderings of the visual information can be synthesized.
In this course we will study this emerging multi-disciplinary field — one which is at the intersection of signal processing, applied optics, computer graphics and vision, electronics, art, and online sharing through social networks. We will examine whether such innovative camera-like sensors can overcome the tough problems in scene understanding and generate insightful awareness. In addition, we will develop new algorithms to exploit unusual optics, programmable wavelength control, and femto-second accurate photon counting to decompose the sensed values into perceptually critical elements.
Goals
- Play with cameras
- Thermal, hyperspectral, high speed, 3D ranging, polarization, line-scan, camera array
- Write a conference-quality paper
- Several conferences coming up: ICCP, CVPR, SIGGRAPH
- Think beyond post-capture computation
- Build your own camera toys
Format
The course will consist of lectures, several homework assignments, one mid-term exam and a final project. Several classes will feature guest talks by experts in the field.
Each week a student will be assigned to take notes during the lecture, and distribute them afterward for class use.
The course emphasizes hands-on hardware and software projects that will progressively build the camera pipeline – this is not a discussion class. We will use several camera elements such as optics (lenses, prisms, apertures, masks), light sources (programmable LEDs and projectors), sensors (high speed, thermal, multispectral, range-sensing) in our projects. However, this is not an optics class; the goal is to learn and build novel imaging devices.
Prerequisites
The course is intended for students with interest in algorithmic and technical aspects of imaging and photography. Successful research in imaging requires a solid understanding in algorithms as well as technologies.
Familiarity with imaging, camera techniques, applied optics, linear algebra and signal processing will be helpful but not necessary. Given the multi-disciplinary nature of the subject, the class will be open and supportive of students with different software and hardware (electronics/optics) backgrounds. We try to keep the mathematical prerequisites to a minimum, but we will introduce material from broad areas at a fast pace.
SIGGRAPH course videos (2007 and 2008) found on Raskar and Tumblin’s Computational Photography page may be helpful.
Related Courses
MAS.532/MAS.132 Camera Culture: Future of Imaging (Prof. Ramesh Raskar)
- Seminar with guest lecturers, discussion oriented
2.71/2.710 Optics (Prof. George Barbastathis)
- Emphasis on Fourier optics and coherent imaging
6.815/6.865 Digital and Computational Photography (Prof. Fredo Durand)
- Emphasis on software methods, graphics, and image processing
Readings
A list of suggested readings will be provided for each class. Suggested readings are also supplied for background on many of the project assignments.
Homework Assignments
Graduate students (enrolled in MAS.531) will complete four homework assignments, while undergraduates (enrolled in MAS.131) will complete the first three of these assignments.
The class material will present concepts with varying degrees of complexity. Each assignment has sub-elements with increasing sophistication, so that students can pick the appropriate level.
Students are encouraged to program in MATLAB® for image analysis. C++, OpenGL and visual programming may be needed for some hardware assignments.
Final Projects
The final project for the class should be novel and cool: hands-on with optics, illumination, sensors, masks. We can provide cameras, lenses, electronics, projectors, etc. Students will produce a conference quality paper describing the project, and we devote the last class period to presentations of the projects, followed by awards for the best projects.
Grading
The credit weights are as follows:
| ACTIVITIES | PERCENTAGES |
|---|---|
| Homework assignments | 40% |
| Final project | 40% |
| Mid-term exam | 20% |
To receive credit, you must attend regularly, complete the project assignment and develop a software or hardware prototype for final project.
Schedule
| LEC # | TOPICS | KEY DATES |
|---|---|---|
| 1 | Introduction and fast-forward preview of all topics | |
| 2 | Modern optics and lenses; ray-matrix operations; context enhanced imaging | |
| 3 |
Epsilon Photography: Improving Film-like Photography (Guest Lecture by Ankit Mohan)
Recent research: Single-shot Multi-domain Camera (guest lecture by anonymous MIT student) |
Homework 1 due |
| 4 | Computational Illumination: dual photography, relighting | |
| 5 |
Lightfields, part 1
Recent research: Retrographic Sensing for the Measurement of Surface Texture and Shape (guest lecture by Micah Kimo Johnson) |
|
| 6 |
Lightfields, part 2
Cameras for human-computer interaction (HCI) Recent research: BiDi Screen (guest lecture by Matt Hirsch) |
Homework 2 due 4 days later Homework 3 out |
| 7 |
IR imaging (guest lecture)
Tomography and 3D techniques (guest lecture) |
|
| 8 |
Wavelengths and colors (guest lecture by Ankit Mohan)
Survey of Hyperspectral Imaging Techniques (guest lecture by Michael Stenner, MITRE) Project ideas discussion |
Final project pre-proposal due
Homework 3 due Homework 4 out |
| 9 |
Computational Imaging: A Survey of Medical and Scientific Applications (guest lecture by Douglas Lanman, Brown University)
“Cameras We Cannot Picture”: a survey of the computational imaging field (guest lecture by Ravi Athale, MITRE) Project pre-proposal presentations (3 min. each) |
|
| 10 |
Mid-term exam
Optics and sensing in animal eyes: what can we learn from successful biological vision systems? (guest lecture by Quinn Smitwick) |
Homework 4 due |
| 11 |
Coded imaging
How to write a paper Wishlist for photography |
Final project proposals due |
| 12 | Final project presentations | Final project report due |
Thanks
The course material has been prepared with slides, discussions and other contributions from many people. Only some of them are listed below, my apologies. Thank you all!
- Jack Tumblin (Northwestern University), Shree Nayar (Columbia University), Amit Agrawal (MERL), Marc Levoy, Bennet Wilburn (Stanford University), Alyosha Efros (CMU), Steve Seitz (UW), Irfan Essa (Georgia Tech), Fredo Durand (MIT), Jingyi Yu (Delaware), Aseem Agrawala (U of Washington), Paul Debevec (USC), Todor Georgiev (Adobe), Hendrik Lensch (MPI).
I also want to thank many who have donated additional equipment, cameras, sensors, optics, software etc.
- Fatih Porikli, Amit Agrawal (MERL), James Davis (UC of Santa Cruz), 3DV, Eddy Talvala and Andrew Adams (Stanford).
