Tomorrow, Boston-based uTest, a startup that crowdsources software quality-assurance projects to a global community of 19,000 freelance testers, is expected to announce the official results of its first “Search Engine Bug Battle.” Xconomy got an early look at the data today, and the folks in Redmond and Bellevue, WA, may not be happy about the results: uTest community members found 321 distinct bugs in Bing, Microsoft’s new search engine, more than the three other tested engines combined.
On the up side for Microsoft, part of the reason Bing fared so poorly compared to the other three engines that testers examined—Google, Yahoo, and Google’s experimental new search engine, Caffeine—may be that Bing is new and high-profile, meaning that testers probed it with extra zeal. Some 85 percent of the 1,100 uTest members who entered the contest chose to make Bing one of the engines they tested. (The same fraction tested Google.) And after testing Bing, 10 percent of the contestants told uTest they liked Bing so much they would make it their default search engine.
Another mitigating factor: Bing “has the youngest code base, compared to Google or Yahoo, so you’d expect to see more bugs,” Matt Johnston, uTest’s vice president of marketing and community, points out. “They’re also introducing features that are new to the search space, including multimedia features, visual search, interactive roll-over features.”
In other words, Bing has more stuff for aggressive software testers to break. Just today, for example, Microsoft unveiled a new “visual search” feature at Bing: users can now initiate and refine their searches by clicking on images, at least in specific categories such as “U.S. politicians.”
uTest’s bug battles are quarterly contests in which the company’s freelance members compete to discover bugs in popular software applications. uTest hands out cash prizes to the testers who find the most. It’s the fourth such battle for uTest, and the first one to focus on search engines. (Previous battles focused on Web browsers, social networking sites, and Twitter applications.)
uTest members not only found more bugs in Bing—321, compared to 130 for Google—but those they did find were more urgent. Contestants described 60 percent of the Bing bugs as either “high severity” or “showstoppers”—those in need of immediate attention—whereas only 8 percent of the Google bugs were showstoppers. A similar percentage of Yahoo’s 70 bugs were classified as showstoppers (10 percent).
The company followed up on the bug competition by surveying members about their search engine experiences. Alas, Bing suffered here too. Asked what quality is most important in a search engine, 71 percent of the uTest contestants answered
