If you mention the 'deep web' in polite company, chances are, if anyone's familiar with it at all, they'll have heard about the drugs, the hit men, and maybe even the grotesque rumors of living human dolls. But there are far more services available through the deep web that aren't illegal or illicit, that instead range merely from the bizarre, to the revolutionary, to the humbly innocuous.
We're talking about websites for people who like to spend their spare time trawling underground tunnels, to websites for people who literally are forced to spend their time in underground tunnels because of the oppressive dictatorial regimes they live in. Then there's a whole lot of extremely niche material—think unseemly book clubs and spanking forums—that has for various reasons been condemned by society.
But first, if you're a member of that polite company that shrugs at its mention, we'll need a working definition. BrightPlanet, a group that specializes in deep web intelligence, simply defines it as: "anything that a search engine can't find." That's because search engines can only show you content that their systems have indexed; they use software called "crawlers" that try to find and index everything on the web by tracking all of the internet's visible hyperlinks.
Inevitably, some of these routes are blocked. You can require a private network to reach your website, or can simply opt out of search engine results. In these cases, in order to reach a webpage, you need to know its exact, complex URL. These URLs—the ones that aren't indexed—are what we call the deep web.
Although its full size is difficult to measure, it's important to remember that the deep web is a truly vast place. According to a study in the Journal of Electronic Publishing, "content in the deep Web is massive—approximately 500 times greater than that visible to conventional search engines." Meanwhile, usage of private networks to access the deep web is often in the millions.
In 2000, there were 1 billion unique URLs indexed by Google. In 2008, there were 1 trillion. Today, in 2014, there are many more than that. Now consider how much bigger the deep web is than that. In other words, the deep web takes the iceberg metaphor to an extreme, when compared to the easily accessible surface web. It comprises around 99 percent of the largest medium in human history: the internet.
Those mind-bending facts aside, let's get a few things straight. The deep web is not all fun and games (weird, illegal, or otherwise). It's full of databases of information from the likes of the US National Oceanic and Atmospheric Administration, JSTOR, NASA, and the Patent and Trademark Office. There are also lots of Intranets—internal networks for companies and universities—that mostly contain dull personnel information.
Then there's a small corner of the deep web called Tor, short for The Onion Routing project, which was initially built by the US Naval Research Laboratory as a way to communicate online anonymously. This, of course, is where the notorious Silk Road and other deep web black markets come in.
Again, that's what you'd expect from a technology that was designed to hide users' identities. Much less predictable are the extensive halls of erotic fan fiction blogs, revolutionary book clubs, spelunking websites, Scientology archives, and resources for Stravinsky-lovers ("48,717 pages of emancipated dissonance"). To get a better idea of the non-drug-and-hit man-related activities one might find on the deep web, let's take a look at some of the most above-board outfits just below the surface.
Jotunbane's Reading Club is a great example, with the website's homepage defiantly proclaiming "Readers Against DRM" above the image of a fist smashing the chains off a book rendered in the style of Soviet propaganda. Typically, the most popular books of the reading club are subversive or sci-fi, with George Orwell's 1984 and William Gibson's Neuromancer ranking at the top.
The ominously named Imperial Library of Trantor, meanwhile, prefers Ralph Ellison's Invisible Man, while Thomas Paine's revolutionary pamphlet from 1776, Common Sense, earns it own website. Some of its first lines aptly read, "Society is produced by our wants, and government by our wickedness; the former promotes our happiness positively by uniting our affections, the latter negatively by restraining our vices." Even the alleged founder of Silk Road, the Dread Pirate Roberts, started a deep web book club in 2011.
So, it seems pretty clear that deep web users like to dabble in politics, but that's far from the whole picture.
Alongside the likes of "The Anarchist Cookbook" and worryingly-named publications like "Defeating Electromagnetic Door Locks," you'll also find a surprisingly active blog for "people who like spanking," where users lovingly recall previous spanks. There's another website with copious amounts of erotic fan fiction: One story called "A cold and lonely night in Agrabah" tells of a saucy tryst with the Jungle Book's lovable Disney bear Baloo, meanwhile Harry Potter is a divisive wizard; some lust over his wand, others declare themselves "anti-Harry Potter fundamentalists."
Alongside the likes of The Anarchist Cookbook and worryingly-named publications like "Defeating Electromagnetic Door Locks," you'll also find a surprisingly active blog for "people who like spanking."
At times, you do wonder if some of the content you come across needs to be on the deep web. A website called Beneath VT documents underground explorations below Virginia Tech, where adventurers frequent the many tunnels that support the university's population of over 30,000 students and 1,000 faculty members. Its creators anonymously explain: "Although these people pass by the grates and manholes that lead to the tunnels every day, few realize what lies beneath."
It's not as though you can't find a plethora of these types of sites of the surface web, illegal or otherwise. But it seems that the deep web offers a symbolic, psychological solace to the users. In practice, the deep web is home to a mix of subcultures with varying desires: all looking for people like them. Beneath VT is one example, but others even offer 24-hour interaction, like Radio Clandestina, a radio station that describes itself as "music to go deep and make love". That's not exactly the kind of tagline you'd see on NPR.
Dr. Ian Walden, a Professor of Information and Communications Law at London's Queen Mary University, explained that the attraction of the deep web is its "use of techniques designed to enable people to communicate anonymously and in a manner that is truthful. The more sophisticated user realizes that what they do on the web leaves many trails and therefore if you want to engage in an activity without being subject to surveillance." He continued, "the sense of community is often what binds these subcultures, in an increasingly disparate and disembodied digital world."
Although the deep web also has a powerful liberating potential, especially since the recent NSA revelations have brought the extent of government surveillance into sharp focus. Surfing along its supposedly safe corridors gives you a strange, exhilarating sensation; probably not unlike how the first internet users felt a quarter of a century ago. Professor Walden has argued that the deep web was vital in the Arab Spring uprising, by allowing dissidents to communicate and unite without being detected. Many of the videos filmed during the Syrian revolution in 2011 were first securely posted on the deep web before being transferred to YouTube.
He points out that "in jurisdictions where political defence is stamped on, social media is not particularly going to help political protest, because it can be quite easy to identify the users." The situation in Turkey earlier this year, for example, saw Prime Minister Erdogan ban the use of Twitter in the country. So instead, Walden suggests, the deep web "allows communication in the long term and in a way that doesn't expose your family to a risk."
It is telling that if the deep web did have a homepage, it would probably be the Hidden Wiki, a wiki page that catalogues some of the deep web's key websites, and that is outspokenly "censorship-free." Its contents give an insight into how these anonymous processes work: the infamous Wikileaks site is hard to miss, but there's also the New Yorker Strongbox, a system created by the magazine to "conceal both your online and physical location from us and to offer full end-to-end encryption" for prospective whistleblowers. Whereas, Kavkaz, a Middle Eastern news site available in Russian, English, Arabic and Turkish, is an impressive independent resource.
"The sense of community is often what binds these subcultures, in an increasingly disparate and disembodied digital world."
Perhaps because the deep web plays host to many of the digitally marginalized and avant-garde, it has also become a hotbed for media innovations. Amber Horsburgh, a digital strategist at Brooklyn creative agency Big Spaceship, spent six months studying the many techniques used in the deep web, and found that it pioneered a lot of innovations in digital advertising.
Horsburgh claims, "As history tells us, the biggest digital advertising trends come from the deep web. Due to the nature of some of the business that happens here, sellers use innovative ways of business in their transactions, marketing, distribution and value chains."
She cites examples of Gmail introducing sponsored email; the social advertising tool Thunderclap, which won a Cannes Innovation Lion in 2013; and the wild success of the native advertising industry, which will boom to around $11 billion in 2017. According to Horsburgh, "each of these 'cutting-edge' innovations were techniques pioneered by the deep web." Native advertising takes its cues from the "astro-turfing" used by China's 50-cent party, where people were paid to post favorable comments on internet forums and comments section in order to sway opinion.
Ultimately, this is the risk of the deep web. "Your terrorists are our freedom fighters," as Professor Walden puts it. In parts, it offers idealism, lightheartedness, and community. In others, it offers the illegal, the immoral, and the grotesque. Take the headline-grabbing example of Bitcoin, which has strong ties with the deep web: It was supposed to provide an alternative monetary system, but, at least at first, it mostly got attention because you could buy drugs with it.
For now, at least, it's heartening to know that some people choose to use the anonymity offered by the deep web to live their mostly harmless—albeit, at times, extremely weird—lives in peace. To paraphrase French writer Voltaire's famous saying: "I may disapprove of what you say, but I will defend to the death your right to make erotic fan fiction about my favorite childhood Disney characters."