To understand the deep web, imagine the Internet as an iceberg. The visible portion of the iceberg is just a small part of the iceberg; most of the ice exists beneath the water’s surface. There is a visible part of the Internet, known as the surface web or clear web, that is easily found and accessed through standard web browsers. But just like the tip of the iceberg, the surface web makes up only a small part of the overall picture—around 6 to 10% of the Internet. Beneath the surface Internet, you’ll find the deep web, which makes up the bulk of the Internet and cannot be found on search engines.
IT security consultant, Earl Foote with Nexus IT in Salt Lake City shares insights into the world of the Dark Web.
Search engines send out robots called crawlers to learn about sites and add that information to databases in a process called indexing. If you want your website to show up on Google, you may use a special instruction file and metadata to attract the crawlers and provide data as part of search engine optimization. But if you don’t want your website indexed by search engines, you can instruct search engine bots to ignore a page or even your entire website. Only people with the URL can see the page. Any non-indexed page is part of the deep web, and some pages on a domain may be indexed while others aren’t.
Table of Contents
Is The Deep Web the Same as the Dark Web?
The dark web is a part of the deep web consists of criminal activity such as child porn, drug trade, and identity theft. The infamous criminal marketplace known as The Silk Road once existed on the dark web. All websites use IP addresses that are hidden by encryption. You need special software, such as the Tor browser, to access the dark web. People who use the dark web aim to conceal their activities from authorities and governments. Some people count on this privacy when talking about health issues related to drug use on dark web forums, and others may search the dark web to see if any of their personal information has been shared on it.
However, while the dark web might be part of the deep web, the deep web is more than just the dark web. The deep web includes any content that isn’t indexed by search engines, and there are plenty of reasons why you might block indexing. For example, medical records, library content, financial accounts, and government websites all exist on the deep web. Additionally, web developers and designers might keep sites on the deep web while working on them before making them visible. And you may block search engines from crawling pages behind the scenes of your website, including administrative dashboards. Even the dark web has legitimate uses. Some people rely on encrypted browsers to access mirrors of popular websites such as Facebook that may be banned in their home countries.
You don’t need special software required to access the deep web. You’ve most certainly logged onto a web page that is part of the deep web without realizing it. A good indicator that a site is part of the deep web, aside from it not appearing on search engines, is that it requires authentication to access. You need to log in to view your bank account balance or medical records and must pay to access academic journals. For sites like these, existing on the deep web is one more level of security to prevent bad actors from access data that doesn’t belong to them.
Keeping some data on the deep web can protect your organization and clients if you do it right.
