IBM's Watson squares off with Carnegie Mellon and Pitt students

Many members of the Carnegie Mellon community watched in February as Watson, IBM’s latest supercomputing project, beat Jeopardy! greats Ken Jennings and Brad Rutter on national television by a very convincing margin. This feat displayed just how far the computer science and artificial intelligence community has come in developing machines that can recognize natural human language, as Watson took on the tricky, witty language used on Jeopardy! with ease. Last week, Watson and IBM researchers came to Pittsburgh to host a full-day event of lectures and Jeopardy!-like competitions with students from Carnegie Mellon and the University of Pittsburgh.

“It’s appropriate that today’s event be held here [in Pittsburgh], the home of a number of the early visionaries and pioneers in computer science and artificial intelligence,” Ronald Larsen, dean of information sciences at Pitt, said in his opening remarks. It was the first university symposium for Watson, with lectures from IBM executives about the breakthrough technology and its numerous applications.

David Ferrucci, an IBM research staff member and leader of integration for the Watson project, spoke about the long road the team had traveled to reach this point, the inevitable growing pains, and, of course, the joy of seeing Watson in action.

One of these growing pains involved the infamous Final Jeopardy! question in which the category was “U.S. Cities,” with the clue, “Its largest airport is named for a World War II hero; its second largest for a World War II battle.” Watson incorrectly answered, “What is Toronto??????,” the multiple question marks indicating its overwhelming uncertainty.

“In Jeopardy!, categories don’t indicate with any strong confidence what the answer might be,” Ferrucci explained. “We have answers that might have really nothing to do with the category. Watson learned this.” For example, there may be a category titled “Authors,” but some of the answers to questions in that category may be unrelated — such as “What is the Book of Job?” or “What is Romania?” — and obviously not authors.

“We could get some signal that Toronto was a U.S. city because they have an American baseball league [and] people refer to it as an American city,” Ferrucci said. “Watson wasn’t determined to find the U.S. city — because it had learned that the category doesn’t necessarily require that.” Interestingly, something that was not displayed during the Jeopardy! broadcast was that Chicago, the correct answer, was Watson’s next best guess, ranking at 11 percent confidence. Watson actually chose Toronto because it had the highest confidence score, a mere 14 percent.

As a demonstration of Watson’s abilities, three Carnegie Mellon students and three Pitt students squared off against the system in a Jeopardy!-like contest. All three students in the Pitt team were members of their university’s quiz-bowl team, though the Carnegie Mellon team students were not.

The first round began with Pitt answering a question correctly in the category “Technologies of the Past,” followed by the Carnegie Mellon team answering a question correctly from the same category. However, Watson buzzed in for the third question, answered it correctly, and maintained its lead for the remainder of the game.

Watson was extremely fast, most often buzzing in first to answer questions and leaving the student teams in the dust. However, this changed during one of the categories in the Double Jeopardy! round. Questions from one of the categories, “The Northernmost Capital City,” displayed three different cities, and the contestants were asked to choose the northernmost capital of the three. Both student teams figured out that if they buzzed in as soon as they were allowed to, despite not immediately knowing the answer, the 33 percent odds of answering correctly were appealing. As such, the majority of the questions in that category were answered correctly by the two student teams by beating Watson to the buzzer.

However, at the end of the game, Watson remained triumphant, with a final score of $52,100, defeating Pitt with $12,937 and Carnegie Mellon with $7,463.

Despite the loss in the demonstration game, the student contestants seemed to have had a good time. “It was pretty cool to be the first college students to be able to [compete] against Watson. We knew we had no chance of winning, but it was still fun,” said Will Zhang, a junior computer science major and a member of the Carnegie Mellon team that faced Watson.

Connor Fallon, another member of the Carnegie Mellon team, expressed amazement at the power of Watson’s technology, but also joked about its lack of manners. “Watson has no sense of humor.... He’s rude. I wouldn’t invite him to dinner.”