The camera that sits atop a tripod in a Gates Computer Science building office befits its tongue-in-cheek moniker. Its body, fashioned from plates of black acrylic held together with screws, is boxy and ungainly. Its innards are assembled from mismatched components—a Texas Instruments "system on a chip" processor, a 5-megapixel image sensor used in camera phones, a Canon EOS lens—some scavenged from dead cameras. Colored wires dangle out of its side. Its creator, Marc Levoy, admits the device is ugly. Hence the name: Frankencamera.
But the beauty is in what it can do: produce images that no other single camera can. Photographs that pop with vibrant, hyperreal hues, regardless of lighting conditions. Pictures without motion blur, even when taken with shaky hands. Images that can be refocused after the shot has been snapped. Levoy, a professor of computer science and electrical engineering at Stanford, wants to fundamentally overhaul digital photography. And this clunky camera is the key.
A traditional single lens reflex (SLR) camera creates an image when light from a scene enters the optics and alters the chemical makeup of the film inside. Skilled film photographers can adjust the focal length of the lens, the aperture (size of the opening through which light enters), and the shutter speed (duration the film is exposed to light), before taking the picture. But once they press the button, they're more or less stuck with the image in terms of focus and lighting characteristics.
Images shot with a digital camera are far more malleable, yet today's commercial cameras are similar in many ways to their film forebears. Electronic sensors detect light entering the lens and record the image as a series of ones and zeros on a chip. One major difference is that digital cameras use microprocessors—mini computers that control the display and the functions of the camera. But Levoy believes they can do so much more.
The first time he picked up a digital camera in the late 1990s, Levoy saw the future of photography. "I wanted it to do something different," he says. "I think any computer scientist who picked up a [digital] camera would have thought, 'Hey, you can program this thing.'"
Sadly, the reality was that you couldn't. Even now, for all the dazzling features manufacturers have introduced in the past few years—cameras that can zero in on smiles, snap a picture when a person enters the frame, or take a series of rapid shots so the best one can be chosen later—consumer cameras are "closed" platforms, their functions essentially baked into the hardware and software. The key to truly improving photography, Levoy's thinking goes, is to open up the camera's controls.
The Frankencamera architecture runs Linux, a standard open-source operating system that lets any programmer tinker with the digital inner workings of the device. Application programming interface (API) software controls and synchronizes —down to the microsecond—the chip that records an image and the chip that processes the image, as well as external hardware like lenses and flashes. When new programs are loaded onto the camera, they can expand the camera's menu offerings, or simply work behind the scenes to make a picture better without any extra fiddling around.
The office where doctoral students Andrew Adams, MS '06, and Eino-Ville "Eddy" Talvala, MS '05, work doesn't have the best light for portraiture, but it's ideal for showing off the Frankencamera's capabilities. Adams's desk sits near a bright window—with a normal camera, taking a picture in which both Adams and the details outside were visible would be impossible. However, when Talvala aims the camera toward Adams and takes the shot, the exposure is just right: The bright daytime scene through the window is perfectly illuminated, as is Adams's smiling face.
The way the Frankencamera achieves this is relatively straightforward, explains Levoy. A few pictures are taken in rapid succession with different exposure settings. Then an algorithm averages them together. The process is automated on the device itself so that the user can see the result instantly. To get that same sort of balanced image, a person using a standard camera would need to manually adjust the exposure time between shots and then manipulate the pictures later with photo-editing software such as Photoshop.
Similarly, by capturing several frames with different focus settings and then combining them in various ways, it is possible to generate a picture in which both background and foreground are in focus, or in which the focus can be shifted after the fact.
While the original Frankencamera prototype (the current model is actually the second generation, dubbed F2) is completely self-contained and portable, it weighs about 4 pounds, making it impractically burdensome to lug around for long periods. So the researchers also have modified the camera interface on a Linux-based Nokia N900 mobile phone to make it programmable. In addition, they've added an external gyroscope to supplement the phone's built-in accelerometer. Software add-ons make it effortless to capture crisp, colorful pictures in low light or in front of a bright background—even when taken by an unsteady hand.
To wit: In Adams's and Talvala's office, the dark underside of a desk comes through clear, even when Talvala's hands shook slightly during exposure. Levoy explains that the camera phone took a burst of 10 quick shots over 3 seconds. Because people's hands aren't actually shaking all the time, some frames are likely to be less blurry. Sure enough, one of them looks significantly better than the others. By analyzing data from the motion sensor, the camera picks the clearest image of the bunch.
In alternate years since 2004, Levoy has taught a Stanford course in which students consider how common problems in photography might be solved computationally—a sort of field test for the Frankencamera project. In the most recent iteration of the class, the first assignment was to write a new autofocus algorithm for the N900. Usually, autofocus is about a second-long process in which the camera estimates the distance to the subject and adjusts the lens accordingly. But students in the class came up with an idea that cut that time in half by speeding up the sweep.
These examples are only the beginning, says Levoy. Cameras could become the new smart phones, in the sense that third-party developers could create apps to extend their capabilities. And smart-phone cameras, with their constant connection to the Internet and access to location information, could even learn to take better pictures. If a camera has access to all the photos you've taken previously, it could more accurately set the exposure to balance the color of a picture of your black dog, regardless of the ambient lighting. Or, when you aim your lens at the Eiffel Tower, the camera might peruse image databases like Flickr to suggest vantage points that are less prosaic.
Concepts like these have been kicking around for a number of years. In 2004, Levoy coined the term "computational photography," and after co-organizing a symposium in 2005, a new field was born. But until the Frankencamera, there had been no physical platform on which to test the ideas that researchers were developing. So the notions remained trapped in academic papers and on optical lab benches with no hope of ever reaching consumers. "We were afraid that this would strangle the field," says Levoy. "And if we couldn't affect industry, why were we wasting our time?"
The mobile-phone industry has been the first to collaborate with the Frankencamera project (Nokia was the group's first sponsor) and Google and Apple also have expressed interest, Levoy says. If his track record is any indication, the impact of Frankencamera-inspired research could be game-changing. Levoy's Digital Michelangelo project, in which the artist's statue of David was swept with laser beams to produce the largest geometric 3D computer model of a scanned object, has launched thousands of research projects. And if you've ever used the paint bucket feature in Photoshop or looked at a location using Google Street View, you're using algorithms that Levoy had a hand in developing.
In July, he presented a paper about the Frankencamera platform at SIGGRAPH, an annual conference and exhibition that showcases the state of the art in computer graphics and interactive technologies. He also has made the source code for the modifications to the Nokia N900 freely available, so that anyone with the phone can open up the camera's controls and start playing. His group has been awarded a National Science Foundation grant to distribute course materials based around the N900 and possibly also build and distribute the next-gen F3 model Frankencamera to other universities.
Researchers in computational photography are excited about the prospect of having a camera with which to experiment. Ramesh Raskar, professor of media arts and sciences at MIT, says that Levoy is essentially expanding the toolset of photographers. "Right now photography is about one brush on canvas," he says. "But once you give individuals control over timing, wavelength, aperture and motion, it's like giving freedom to an artist. This will hopefully create the next generation of digital art."
Other educators see the Frankencamera as the perfect teaching tool. "I actually think it's a sneaky way of introducing the concept of programming to people who might never have considered it," says Hanspeter Pfister, a professor of the practice of computer science at Harvard. As photographers who worked with film often became chemists in order to modify their art, so too photographers who work with digital cameras could become programmers, he says.
For Levoy, the goal is not to turn every shutterbug into a Steiglitz, but rather to share the possibility of creation with fellow researchers and computational photography enthusiasts. Then the ideas behind the Frankencamera could take on a life of their own.
KATE GREENE is a science and technology journalist based in San Francisco.
