It is unable to ace the test for a variety of reasons, Greaves says. Sometimes it lacks enough specialized knowledge dumped into its knowledge base, sometimes its reasoning algorithm is too weak, or there’s a bug in the code, or the user does a poor job in translating the natural-language form of the question into the variety of English the machine can understand.
The example he gave is when a question is phrased like “an alien drops a rock from the top of a 100-foot high wall. How long does it take to hit the ground?” It’s a standard physics question, but the machine can’t answer it directly, because extraneous detail like the fact that an alien is involved will throw it off course. No current system can reliably understand unrestricted English, the way questions are usually phrased in real life. “Our system can’t answer questions like how a teacher asks it. There’s too many semantics involved. It’s beyond the state of the art,” Greaves says.
So the knowledge engineers at Vulcan’s contractors—which include Menlo Park, CA-based SRI International, Boeing’s Phantom Works research center in Bellevue, WA, and Germany-based Ontoprise—train graduate students to phrase the question to the computer in a way it can understand, Greaves says. Sometimes that means being more precise about things that were implicit in the original question, or stripping away extraneous detail.
The new goal is to fine-tune the processes so the machine can get 75 percent of the answers right on the AP exams. There were 12 students involved in the first experiment, and now Vulcan’s contrators will expand the pool to 20 or 30 with the hope of getting a more statistically robust result this time, Greaves says. Project Halo also includes an element of wikis, which harness the power of crowds of Internet users to build up bases of scientific knowledge that the machine can reason through, Greaves says. This has the advantage of tapping the expertise of people around the world who can assemble almanac-style information that computers can later access and use to answer questions. (Think about how Seattle has 594,000 residents, and about 7,085 people per square mile. Enter enough of these factoids, and eventually the machine could answer a question about which cities are most dense, rather than simply refer to an online document that has some data on the subject.)
Vulcan doesn’t say publicly how much it has invested in the project, or exactly what commercial applications it may have in mind. Greaves did suggest, however, that it might be used for educating students, test preparation, or as a research assistant to scientists. He wouldn’t go on record with a prediction on how long it will take to make this software ready for prime-time usage. But he didn’t sound shy about whether the vision of a computer that answers questions is achievable. “We’ll do what it takes to get the Digital Aristotle out there. The goal is to have an impact on the world,” Greaves says.
