Not Your Grandma's Search Engine

Imagine a trawler in the ocean. Its net captures what swims close to the surface. That’s what search engines do.
But the Internet, like the ocean, is deep — widely estimated to be 500 times larger than the space most search engines have managed to index — and to truly explore it, you need specialist tools.
Quick example.

Let’s take India’s claim that it exports software worth $30 million to the US.

To authenticate this, you’d need to cross-check American imports of Indian software. Answers such as these are generated only when you ask for them, from databases; they’re “dynamic” content. Most search engines deal with static content.

Image: lii.org

What you need therefore, is an engine that queries a specific database, and throws up the answer you’re looking for.

Take my hand. Trust me.

Welcome to the Deep Web. Let’s start with Librarians Internet Index . Careful navigation will eventually lead you to the Interactive Tariff and Data Web. A few minutes of figuring-the-system-out later, the answer will come up on your screen — downloadable as a spreadsheet too — completely out of whack with India’s claims.
Get the drift?

If these sites feel intimidating, try Noodle Tools . And if Noodle intimidates you, perhaps you’d better go play in the shallows.

Still with me? There’s more: the Dark Net, a.k.a. the Invisible Net, a.k.a. the Private Net, territory that cannot be browsed even by Deep Web search engines. Where you can ferret out rare books, music and films from fortress-like machines.
Entry is by invitation only, encryption is heavy, and it takes patience to break into
its labyrinths. But once in, no sysadmin, no business entity, no government can stop you from looking for what you want to see.

The gates open at Free Net . Follow instructions, install the software and start looking around.
For the more enterprising and experienced, there’s WASTE . Yanked off the Internet after AOL bought its developer (AOL went on to call it illegal), it was adopted by a coalition of programmers who believe in file sharing and free software. It can now be downloaded from wasteagain.sourceforge.net.

Big tip: Visit Fravia — the father of search on the Web. Now terminally ill, he has stopped updating his site; but his archives reside on searchlores.org , loaded with wit, charm and subversiveness.
Go visit. It’s where you start learning to search

Click here to read on the alternate search engines available to mine the internet.