For 15 years now, we’ve all thought of the World Wide Web as a near-literal web of connections between millions of servers in different locations, with each machine hosting just a tiny slice of the Web’s overall content. But that’s not the new shape of the Web, according to Arbor Networks of Chelmsford, MA. Today, a startling amount of Web content and traffic is controlled by just a handful of large Internet companies.
“As of 2009, 60 percent of all Internet content comes from, or terminates within, just 100 to 150 companies,” says Craig Labovitz, Arbor’s chief scientist. “That’s a very dramatic change in where the data is coming from.”
What that means in practical terms is that if you surf to the website for, say, the Pretty Good Car Company, more than likely the data is no longer stored on servers at Pretty Good itself, but on machines owned by a centralized infrastructure provider that Pretty Good has hired to handle its site, such as Akamai, Limelight, Rackspace, Amazon, Equinix, GoDaddy, or Verizon.
Arbor released data on this and other trends yesterday at the North American Network Operators Group conference in Dearborn, MI. The company makes software that helps companies detect and prevent denial-of-service attacks against their Internet servers.That software is installed on the Internet routers of 70 to 80 percent of the top content providers and Internet service providers in North America, which allows Arbor to collect vast amounts of information about Internet traffic.
Indeed, Arbor’s view of network traffic rivals and in some ways surpasses that of Cambridge, MA-based Akamai, whose “State of the Internet” reports we’ve covered frequently here. Only about 20 percent of global Internet traffic passes through Akamai’s content distribution network.
In a trend that Arbor calls the “rise of the hyper giants,” most Web content and traffic is moving to a small number of very large hosting providers and cloud services companies. The world’s Internet addresses are controlled by about 35,000 network operators, and as recently as 2007, the majority of Internet traffic was smoothly distributed across these operators. But today, 60 percent of all Internet traffic is generated by just 100 companies, according to Arbor, which conducted its study in collaboration with the University of Michigan and the non-profit Merit Network in Ann Arbor, MI.
There is concentration at the very top: 30 percent of all traffic comes from just 30 companies. And Google alone generates about 6 percent of all Internet traffic, Arbor found. Akamai, Microsoft, Limelight, Yahoo, and GigaNews (which hosts Usenet newsgroups) are also on the list of “hyper giants.”
“Basically, 150 to 200 companies are now generating the majority of Internet content, at least as measured by traffic,” says Labovitz. It’s an inevitable and, in some ways, unsurprising trend, given the rising popularity of cloud-based hosting models for both content and software, and in view of the huge investment required to build data centers with the processing and communications capacity to handle today’s most popular forms of content, especially bandwidth-hogging video. But one implication, of course, is that outages and other snafus in a single location can affect many more Internet users all at once.
“Ensuring availability used to mean backing up your mail server or your laptop,” says Labovitz. “Nowadays, what does it mean if all your e-mail is on Google and Google is down for the day? If it wasn’t true before, it is very quickly becoming true: the network is the computer.”
