SciTech

Photo-editing system manipulates objects three-dimensionally

Original photo of an origami crane. (credit: Yaser Sheikh) Original photo of an origami crane. (credit: Yaser Sheikh) Photo after 3-D object manipulation. (credit: Yaser Sheikh) Photo after 3-D object manipulation. (credit: Yaser Sheikh) Photo of laptop prior to photo editing. (credit: Yaser Sheikh) Photo of laptop prior to photo editing. (credit: Yaser Sheikh) Photo after laptop was rotated in photo-editing system. (credit: Yaser Sheikh) Photo after laptop was rotated in photo-editing system. (credit: Yaser Sheikh) Yaser Sheikh, associate research professor of robotics. (credit: Yaser Sheikh) Yaser Sheikh, associate research professor of robotics. (credit: Yaser Sheikh) Natasha Kholgade, graduate student and lead author of the study. (credit: Natasha Kholgade) Natasha Kholgade, graduate student and lead author of the study. (credit: Natasha Kholgade)

It’s effortless for us to pick up an object and manipulate it in 3-D in real life, but what if we could twist and turn objects in a photograph just as easily?

Yaser Sheikh, associate research professor of robotics, and Natasha Kholgade, a graduate student in the Robotics Institute, have created a photo editing system that allows users to move objects three-dimensionally, turning or flipping them in ways that reveal parts of the object that were not even in the original photograph. Other members of the group include Tomas Simon, a Ph.D. student in the Robotics Institute, and Alexei Efros, a former Carnegie Mellon faculty member.

“We started the project about three years ago, when we realized that while interacting with photographs in editing software, there were several 3-D manipulations that we intuitively felt like doing … that were simply not possible in photo-editing software,” Kholgade said. While users can scale, rotate, and move objects in current photo-editing software, Kholgade’s new system allows users to flip and rotate objects in all different directions, knock them over, or pick them up and place them somewhere else in the photograph — all while maintaining the photograph’s realism.

But how is it possible to turn an object to a side that wasn’t even present in the original photograph? The team’s system takes advantage of the 3-D models of common objects that are readily available online. “Today, there are large collections of 3-D models available on public repositories such as 3D Warehouse and TurboSquid,” Kholgade said.

While this method may not be able to cover all the bases right now, the inevitable growth of these online databases will soon solve this problem. “We can expect these repositories to keep growing at an exponential rate, since 3-D printing and scanning technologies are becoming more and more ubiquitous, and several items such as furniture, vehicles, toys, and household equipment are becoming standardized,” Kholgade added. In the current version of the system, the user finds the corresponding 3-D model needed for an object in a photograph by using a word search.

Despite the wide availability of 3-D models online, however, there are always inherent differences between online models of objects and the actual object in a given photo. Kholgade explained that the main differences between online models and an actual object in a photograph are their colors, textures, and shape. “For instance, you will never find that discolored banana in your fruit basket, or your specific dusty, worn out hat in the 3-D model repository,” Kholgade said. To account for this, the system provides a user-guided approach to align the shape of the 3-D model to the object in the photograph. It then factors out scene lighting in the photograph and completes the appearance of a surface that was not in the original photograph by drawing symmetries over an object.

In addition to just turning or flipping an object in 3-D, this system uses the same algorithms to create realistic animation. “Our system is tied to standard animation and modeling software, through which the user can create an animation for an object. Our system then deforms the object in accordance with the animation, while providing plausible shading and shadows, and remaining faithful to the original photograph,” Kholgade said. The system is also compatible with paintings and historical photos, as long as they are converted into an electronic format.

Besides just being a game-changer for photo-editing software, the team’s system has real-world applications in furniture rearrangement while putting personal homes on the market, three-dimensional viewing of merchandise in online shopping catalogs, or even in connecting physical simulations of events or anatomical processes to photographs for forensics or medicinal purposes.

In the future, the team hopes to automate the search and retrieval of online 3-D models, as well as the alignment of 3-D models to actual objects in photographs. This system has flipped the world of photo-editing 180 degrees.