top of his page. That code tells the end user’s Web browser that an experiment is running and that, as the page is loading, the original HTML or graphics should be swapped with whatever new elements the owner specified within Optimizely’s editing interface—a larger headline, for example.
Siroker’s explanation is pretty straightforward: “The page loads just as it normally would, but once the browser loads our Javascript file, it says, ‘This person is a new visitor, so I am going to bucket them into one of the variations. I want this visitor to be in Variation 2 and that means we’re trying a new headline, so we are going to find the unique identifier for that headline and replace it with the new copy you wrote. And with every subsequent action, let’s track it to record progress toward the goals.'”
If you’re having trouble visualizing all that, I recommend watching Optimizely’s 4-minute introductory video. Suffice it to say that there’s enough money at stake in the worlds of e-commerce and online marketing to make a subscription to Optimizely worthwhile for many customers. The service ranges from $17 per month to $359 per month or more, depending on the number of visitors being “bucketed” into Optimizely’s tests.
The startup’s first big client, though, was actually a pro bono account: the Clinton Bush Haiti Fund, an initiative to support rebuilding in Haiti after the devastating January 2010 earthquake. “Their goal was about getting people who come to the site to donate more money,” says Siroker. “We did several variations. At the bottom of the donation page, instead of saying ‘Submit,’ we changed the button to say ‘Support Haiti,’ or we increased the font size or the layout or included images of the victims. The net impact was an improvement of about 10 percent on the dollars per page view. By the time the experiments were done, they had raised $11 million, about $1 million of which could pretty directly be tied to these improvements.”
That’s the power of data for you. Of course, there are limits to A/B testing. For one thing, the end result of a test will only be as good as the elements you were testing in the first place. “I don’t think you can start with a blank page and A/B-test your way to success,” says Koomen. “It requires creativity and initiative.”
Google itself has often been criticized for taking data-driven decision making to extremes; one visual designer for the company famously quit in 2009 after tiring of a culture where it was standard procedure to test 41 shades of blue for the Google toolbar to see which performed best. And here’s a telling fact—the Obama White House isn’t using A/B testing tools on the whitehouse.gov website. “Using technology to run a campaign is very different from using technology to govern,” says Siroker. “One challenge [with whitehouse.gov] is figuring out what to optimize for. Do you want people to watch this video, or share something on Facebook?” (Or maybe just learn something about how their government works?)
Online businesses and Web designers are closer to Optimizely’s sweet spot, and Koomen says the company has found enough users that it’s been profitable (or at least ramen profitable) since before it exited Y Combinator. The startup has raised $1.2 million in angel backing from the usual suspects, including Sam Altman, Paul Buchheit, Ron Conway, Chris Dixon, Steve Huffman, Nils Johnson, Mitch Kapor, Ashton Kutcher, Ariel Poler, Naval Ravikant, Aydin Senkut, Ram Shriram, Joshua Schachter, and Brian Sugar. It’s now got nine employees, all working from the startup’s stylish loft adjacent to San Francisco’s South Park startup haven, and there’s enough revenue coming in the door that the company doesn’t need to think about raising more investment, Koomen says. The startup’s focus right now is in “customer service and retention, learning our sales cycle, and building out the product,” he says.
Then there’s the 2012 presidential election coming up, when, it’s safe to say, online fundraising and A/B testing will be in even wider use. “I noticed that for a lot of campaigns [in 2008], their optimization strategy was just to copy what Obama did,” says Siroker. “Candidates always like to do the thing that worked last time. Plus, if you really want to push the limit and improve on what’s there [on a website], the only way you can do that is optimization.” So don’t be surprised if the Donate Now button on your favorite candidate’s website is in a different spot, or a different shade of blue, every time you visit. It just means you’ve been bucketed.
Here’s Optimizely’s introductory video.
