How big is The Web?
Adam Rifkin stashed this in The Web
I'm trying to figure out how big the public web is, and how big the "dark web" or "deep web" is.
As of a year ago, the public web grew by 150k URLs each day.
It's probably growing even faster now.
Cisco estimates of how big the Internet is: http://techland.time.com/2011/06/01/how-big-is-the-internet-spoiler-not-as-big-as-itll-be-in-2015/
Cisco, which releases a Visual Networking Index forecast about the scale of web traffic every year, estimates that by 2015, 966 exabytes of information will be consumed annually by the internet – an increase of 200 exabytes over the estimate for 2014, which is itself more than the amount of all 2010 information consumption online.
By comparison, it took until 2004 for monthly internet traffic to pass 1 exabyte for the first time (For those wondering what an exabyte is, it’s 1,000,000,000,000,000,000 bytes; it was once estimated by UC Berkeley that “all words ever spoken by human beings” could be stored in 5 exabytes, although that would likely have to be in text format).
Google had a trillion URLs collected as of 2008: http://googleblog.blogspot.com/2008/07/we-knew-web-was-big.html
By 2011 there were a trillion actual Web pages, according to Kevin Kelly: http://articles.cnn.com/2011-09-12/tech/web.index_1_internet-neurons-human-brain
About.com calls it the Invisible Web: http://websearch.about.com/od/invisibleweb/f/What-Is-The-Size-Of-The-Hidden-Web.htm
Google's index is between 40 and 50 billion pages: http://www.worldwidewebsize.com/