Like Photoshop, but easier

People always bring back the same couple of things from family vacations: tan lines, empty wallets, made-in-China souvenirs, and — of course — photographs. And no family album could be complete without the requisite duds: Mom’s poorly timed shot of Lake Winnipesaukee, obscured by that boat that sailed into the picture; Dad’s artsy still of Abbey Road at sunrise, looking eerily vacant instead of serene. Hoping to remedy such photo faux pas, researchers at Carnegie Mellon developed two new photo editing techniques that could help the next family album earn its place on the coffee table.

Ditching the sailboat

James Hays, a Ph.D. candidate in SCS’s Graphics Lab, created a system called Scene Completion; he designed the technique around flickr.com. Scene Completion may just be the perfect solution for a botched Winnipesaukee scene — the software makes it easy to fill in a photograph’s holes, whether from physical damage or the removal of an object (an obtrusive sailboat, for example). It considers the colors and textures of the spot surrounding the hole, in addition to contextual factors; it won’t suggest a giraffe to fill a lake, for example.

Using this criteria, the Scene Completion method presents its user with a list of up to 20 images that can fill in the hole. So, Mom can choose to add a kayak, another sailboat, a swimming tourist, etc. Not all of the 20 photographs will fit in the picture (some might not match the lighting, for example), but two or three likely will. “The method has a low success rate,” said Hays. “It needs a lot of data.” The accuracy of Scene Completion increases with the amount of images it can search from.

Populating Abbey Road

For Dad’s deserted Abbey Road, graduate student Jean-François Lalonde created a technique called Photo Clip Art. Lalonde’s method uses images divided into categories, such as people, cars, and trees. “We want to add specific objects,” he said. The program asks its user to click on a photograph’s horizon line, which it uses to define criteria for the images that someone might want to add. At this point, if a user clicks on “car,” the program will show all of its photographs of cars in an ordered list; the ones that closest match the original image’s lighting and camera angle conditions are listed at the top.

Photo Clip Art is especially useful for people interested in modeling projects, such as an architect proposing a building plan or a screenwriter working on a storyboard. An architect, for example, might be interested in seeing how the lawn of a prospective complex would look with people on it, Lalonde explained.

Aside from the specifics, the overwhelming benefit of both photo editing projects is that Mom and Dad can actually use them. “Our goal was to make this not completely automatic but as easy as possible,” Lalonde said. Users need not understand Adobe Photoshop — or even Microsoft Paint — to use the software.


With this kind of accessibility, there’s a risk that editing photographs could become too easy. If Mom and Dad can add a car and subtract a sailboat from every family picture, the entire album could lose its credibility. Moreover, if photojournalists can add, say, an exploding warhead to a photograph of Iraq, it could lead to credibility problems that eclipse even the Jones’s Family Album.

“There’s been reason to be suspicious of photos for a long time,” said Hays. Although the techniques coming from Carnegie Mellon may be new, the results that they achieve have long been possible through tools like Photoshop and other programs. In terms of what you can do, said Hays, “the sky is the limit.”

And in a way, it always has been. One of the earliest photo controversies dates back to 1858, when Henry Peach Robinson published a photograph called “Fading Away.” The picture showed a young girl dying while her family watched and wept, but there was only one problem: It wasn’t real. The people in the photograph were all actors, and, on top of that, Robinson created the image by combining the negatives of five separate photographs.

“The public got really upset. They got duped,” said Charlee Brodsky, a professor in the School of Art. It’s easy to think that new technology in photo editing is going to change photography forever, or at least its credibility. But, as Brodsky explained, the Scene Completion and Photo Clip Art projects are only newer and easier ways to create effects that have been possible for years. “I think that what people might not know is that we’ve had trickster photography for a long time,” said Brodsky. “We had photographic manipulation in the darkroom.”

With Carnegie Mellon’s tools for the technology-deficient, Mom and Dad could become the next Henry Peach Robinson.