Software detects stylistic features

Credit: Adelaide Cole/Art Editor Credit: Adelaide Cole/Art Editor

Carl Doersch, a doctoral student studying machine learning, and his colleagues have developed new graphical software capable of identifying stylistic features of cities. The team, composed of researchers from both Carnegie Mellon and the Institut National de Recherche en Informatique et en Automatique, has published its work in the computer graphics journal ACM Transactions on Graphics.

The process of finding related patterns between images is known as visual data mining. Project collaborator Alexei Efros, an associate professor of robotics and computer science at Carnegie Mellon, pointed out that the science is still relatively new.

“The field of visual data mining is still in its infancy, but I believe it holds a lot of promise,” he said in a university press release. “Our data mining technique was able to go through millions of [images] automatically — something that no human would be patient enough to do.”

The researchers’ full data set actually contained a quarter of a billion images. They were collected from Google Street View.

The group’s program works by parsing through a given set of captured city imagery, identifying the elements of that city’s architecture or scenery that may help distinguish it from other cities. Specifically, the algorithm looks for representative features of a city — features that are both frequent, like trees, and specific, like Paris’ Eiffel Tower.

For example, the algorithm labeled window balustrades, traditional street signs, distinct lampposts, balcony supports, and doors as some of the representative features of Paris. London, on the other hand, sports a combination of neoclassical columns, Victorian windows, and cast-iron fencing, all coming together to give each city a unique look.

But extracting these rare, indicative markers was no easy task. Doersch compared the process to that of finding needles in a haystack.

“It was quite a challenge to find an algorithm that could reliably sift through all the uninteresting [images] without throwing out the needles,” he said.

Despite the challenges, the researchers had considerable success with their algorithm’s accuracy. European cities like Paris, London, Prague, Milan, and Barcelona — which have had distinctive architectural styles for centuries — proved especially responsive to the algorithm. However, there has been some difficulty getting it to work with cities in the U.S.

“When we tried our algorithm on American cities, we found considerably less [features] than in European cities,” Doersch said. “Maybe this isn’t so surprising though, since the U.S. is in many ways a melting pot of cultures. Usually each city has so many different architectural influences that no single architectural style can dominate.”

The researchers presented their findings at SIGGRAPH, an annual conference about advancements in computer graphics, earlier this month. “I spoke to quite a few people afterward who wanted to know more and were interested in taking it further,” Doersch said.

One practical usage for the team’s research includes the potential for a computer-assisted analysis of the spread of certain architectural styles across a given area. Doersch also highlighted another, perhaps more interesting, application.

“Computer animators often have to build models of cities,” he said. “For example, for the movie Ratatouille, Pixar built a model of Paris.”
Doersch explained that to do this, the animators spent a few weeks in Paris carefully studying and taking pictures of the architecture in order to decide what to put in the façades to make them look Parisian.

“Our algorithm could help automate this process, providing a way to organize the collections of reference photos so that the animators can get a more complete picture of the city faster,” Doersch said.