Chicago while everyone else was in Iowa,” Siroker recounts. “When I showed up they had a guy doing blogging and a guy doing e-mail, and a ton of people coming to the website. But they didn’t know what they should be looking at. Joe Rospars, the director of new media, thought the biggest opportunity was to tap into that data. We started with one simple experiment on the splash page that generated a substantial increase in e-mail signups, and in the end it meant big dollars for the campaign.”
But those tests employed two commercial, off-the-shelf A/B testing tools, Google Website Optimizer and Omniture’s Test and Target (now Adobe Test & Target—originally the creation of Jamie and Matthew Roche, who went on to start Bo.lt). Neither was developed with non-programmers in mind. “Both had huge up-front costs in understanding how to use the product,” Siroker says. “Both required you to tag different parts of a page [before testing it], and put a block of script above and below those parts. To run the experiment you had to commit code and go through QA [quality assurance]. The process was what hampered us. We had a hundred brilliant copywriters on the campaign, but we were bottlenecked on engineers to run the experiments.”
That pain provided the initial spark for Optimizely. But Siroker says that the campaign left him feeling a little burned out on Web analytics—so the spark lay dormant for a while.
After the campaign and the transition, Siroker returned to San Francisco, where he and Koomen built an education startup called CarrotSticks. The site offers games where first- through fifth-graders can compete with peers to finish math problems. It’s still online today, but more as a hobby than a business; Koomen says he and Siroker quickly realized that “it’s tremendously difficult to make money in education. Schools are broke, and parents are difficult to reach. The product ended up spreading quite well, and it pays its own server bills, but we decided that there were other ideas that were a lot bigger.”
The story doesn’t come back to A/B testing quite yet. When Koomen and Siroker decided to apply to the Winter 2010 term at Y Combinator, they pitched an idea for a social commerce startup they called Spreadly. The idea was to give people incentives to tweet or post Facebook status updates about purchases they’d just made—tickets to see Justin Bieber, for example—as a tool for marketers. But after getting into YC and spending a month putting together a prototype, the pair quickly saw that “it’s really hard to pay a person to broadcast something to their friends,” Koomen says. “If you think of it in terms of social capital, the amount they are spending by sending something out to their friends is almost never going to match up to the amount you are offering them.”
That’s when Spreadly finally gave way to Optimizely. “At the end of the day, I enjoy data analysis, and that’s something I only realized once I got out of it,” Siroker says. “I was more burned out from the sedentary lifestyle [of the campaign], and no sleep, and sitting in a room for months eating deep dish. But now for the first time, we were working on something that I would want—I was scratching my own itch. Neither of us were parents”—or Justin Bieber fans, for that matter—“but I was the guy who could have used this technology back in the campaign.”
Siroker and Koomen built Optimizely using jQuery, a relatively new technology that uses Javascript to make it easier to change the look or behavior of a Web page on the fly. To use Optimizely’s system, a website owner only needs to insert one line of code at the
