oio11 | The Invisible Web: What It Is, How You Can Find It

The Invisible Web: What It Is, How You Can Find It

By Wendy Boswell
Web Search Expert

Woman Working On Laptop In Kitchen - Cavan Images/Taxi/Getty Images — Cavan Images/Taxi/Getty Images

Updated December 16, 2014.

What is the Invisible Web? Is it some kind of Area 52-ish, X-Files deal that only those with stamped numbers on their foreheads can access? Well, not exactly. The term "invisible web" mainly refers to the vast repository of information that search engines and directories don't have direct access to, like databases. Unlike pages on the visible Web (that is, the Web that you can access from search engines and directories), information in databases is generally inaccessible to the softwarespiders and crawlers that create search engine indexes.

How Big is the Invisible Web?

In a word, it's humungous. Bright Planet estimates the invisible, or deep, web as being500 times biggerthan the searchable, or surface, Web. Considering that Google alone covers around 8 billion pages, that's just mind boggling.

Why Is It Called "The Invisible Web"?

Spiders meander throughout the Web, indexing the addresses of pages they discover.

When these software programs run into a page from the Invisible Web, they don't know quite what to do with it. These spiders can record the address, but can't tell you squat about the information the page contains. Why? There's a lot of factors, but mainly they boil down to technical barriers and/or deliberate decisions on the part of the site owner(s) to exclude their pages from search engine spiders. For instance, university library sites that require passwords to access their information will not be included in search engine results, as well as script-based pages that are not easily read by search engine spiders.

Why Is The Invisible Web Important?

Perhaps you think it would be easier to just stick with what you can find with Google or Yahoo.

Maybe. However, it's not always easy to find what you're looking for with a search engine, especially if you're looking for something a bit complicated or obscure. Think about the Web as a vast library. You wouldn't expect to just walk in the front door and immediately find information on the history of paper clips lying on the front desk, right?

You might have to dig for it. This is where search engines will not necessarily help you, and the Invisible Web will.

Plus, the fact that search engines only search a very small portion of the web make the Invisible Web a very tempting resource. There's a lot more information out there than we could ever imagine.

How Do I Use The Invisible Web?

Fortunately for you and I, there are many other people that have asked themselves the exact same question, and have put together great sites that serve as a launching point into the Invisible Web. Here are some gateways for different subjects:

Humanities

Voice of the Shuttle: Started in 1994, VoS is one of the oldest and largest humanities databases on the Web, offering an exclusive index of content for subject ranging from Anthropology to Technology of Writing.

Specific to U.S. Government

University of Michigan Government Documents Center: You'll find reports, statistics, and other documents from all levels of the U.S. government here. Databases offered include Arts, Health Sciences, Social Sciences, and International Studies.
USA.gov: A red carpet portal into the depths of the United States government's many entities. Includes government jobs, an A to Z list of governmental agencies, and information on finding grants, loans, and financial assistance.

Health and Science

PsycNET: Use this database from the American Psychological Association to find abstracts and entire journals on various psychological topics (fees apply to complete journal entries, abstracts are usually free).
Scirus: A search tool dedicated only to scientific information, this amazing search tool has indexed hundreds of millions of scientific, scholarly documents to aid searchers all over the world in their pursuit of knowledge.
Healthfinder: Professionally vetted information from over one thousand different health databases on the Web. A wide range of resources on a dizzying array of health topics is available here.
RXList: If you're looking for reliable drug information, then this database for prescription drugs, medications, and pill identifications is for you. Offers an A to Z index, image collections, and a dictionary of terms.

Mega-Portals

The University of California, Riverside maintains InfoMine, an incredible resource that at last count included over 100,000 links and access to hundreds, if not thousands, of databases.
The Virtual Library is simple and easy to use, with annotated subject links. I especially appreciate the annotations because it helps rule out extraneous search time.

What About Other Invisible Web Resources?

There are many, many sites that are set up to dig into the Invisible Web.CompletePlanet.com is a directory of "over 70,000+ searchable databases and specialty search engines."

Most of the information on the Invisible Web is maintained by academic institutions, and has a higher quality than search engine results. There are "academic gateways" that can help you find this information. To find nearly any educational resource on the Web, simply type in this search string to your favorite search engine:

site:.edu "subject I'm looking for"

Your search will return with only .edu-related sites. If you have a particular school in mind that you'd like to search, use that school's URLin your search:

site:www.school.edu "subject I'm looking for"

Frame your subject within quotations if it is more than two words; this lets the search engine that you're using know that you want to find those two words right next to each other (for more on basic search techniques, read Web Search 101: The Basics).

The Bottom Line About The Invisible Web

This is just the tip of the iceberg, folks. The links I've highlighted in this article barely begin to touch the vast resources available on the Invisible Web. As time goes on, the Invisible Web will only get bigger, and that's why it's a good idea to learn how to use it now.

Our Expert Recommends

http://websearch.about.com/od/invisibleweb/a/invisible_web.htm

Flat | Top-Level Comments Only

From:

one_versus_all

Неделю не получилось зайти на Дривидс, пока не попробовал кнопку Турбо на Опере =).
Буду через турбу ходить =).
Как НГ праздники? =)

From:

oio11

Праздники - как всегда, слишком быстро проходят:))

Все-таки странно, почему с Дримом пробемы..
наверное браузер надо менять:)
Самый лучший и быстрый - Links
http://links.twibright.com/
https://en.wikipedia.org/wiki/Links_(web_browser)
и даже,кажется, для Windows есть версия
http://portableapps.com/apps/internet/links-portable
opera мне перестал нравится, разве что для просмотра видео пользуюсь

Edited Date: 2016-01-10 12:34 pm (UTC)

From:

one_versus_all

http://craftwr.livejournal.com/79608.html
http://craftwr.livejournal.com/79099.html
http://craftwr.livejournal.com/77130.html
=)

From:

oio11

"ничего личного, просто - бизнес":)
а с учетом встроенной батареи во всех этих "новинках"
разницы большой между ними я не вижу
только что памяти больше
http://craftwr.livejournal.com/79608.html

из комментов - "Плохая ОС может испортить жесткий диск"
Это точно! Было такое, когда новое ядро "экспериментальное" в линуксе попробовала поставить:))
Насчет Виндовс не в курсе
http://craftwr.livejournal.com/79099.html

гном вообще попробовала убрать,
частично мне это удалось:)
http://craftwr.livejournal.com/77130.html

Edited Date: 2016-01-10 01:56 pm (UTC)

From:

oio11

Вот оно что,оказывается ..

https://en.wikipedia.org/wiki/Hummingbird

а я то не могла понять, что с поиском не так
в русскоязычной Википедии даже упоминания не находится:)

Всем Колибри: самое крупное обновление поискового алгоритма Google за 3 года
30 сентября 2013, 07:47 SMM Александр Мельник

SEO-специалисты снова расчехляют свои клавиатуры и затариваются крепким кофе: Google не просто отпраздновал 15 лет, а еще и выкатил новый поисковый алгоритм с преферансом и поэтессами с распознаванием смыслов и новым анализом запросов.

Новинка называется Hummingbird (Колибри) и направлена на "очеловечивание" поиска. По сути, теперь Гугл должен научиться (или уже научился, как утверждают разработчики) распознавать смыслы в сложных запросах, а не просто реагировать на указанные ключевые слова. Колибри якобы учитывает все контексты поискового запроса ..
https://vc.ru/p/hummingbird

.. Новый алгоритм гугла
Мне кажется, что это всего лишь девальвация таких доменов, т.е. раньше они были в приоритете и Google их определял как наиболее релевантными, а теперь они просто заняли свои реальные позиции. Как-то так. ..
http://searchengines.guru/showthread.php?t=744115&page=9

"очеловечивание", да
это заметно..:))
машина беспристрастна, в отличие от некоторых людей:)

Searcher Behavior
Posted on April 2, 2013 by Greg

A new study from iAquire and SurveyMonkey highlights searcher behavior — not professional searchers but the general web searchers. While the focus is on marketing for retail sites, it has some useful for informational findings as well. I found the following sad but not surprising:
40% of searching activity is shopping (e-commerce), but
65% don’t click on ads, or at least say they don’t since
50% of those who do click on ads can’t differentiate the ads from regular search results.
On the positive side, I was surprised (but pleased) to hear that 90% of users will view the second page of search results. ..
http://searchengineshowdown.com/2013/04/searcher-behavior/

Edited Date: 2016-01-10 01:55 pm (UTC)

From:

oio11

что-то в этом роде:))
www.youtube.com/watch?v=EOMstKCg0d8

From:

one_versus_all

Эх, яблочко... =):
http://craftwr.livejournal.com/80708.html

Взялся после 6-летнего перерыва досмотреть сериал Lost =). Подстроенная авиакатастрофа, постановка крушения, древнеегипетская тематика, архитектура, кресты-анкхи, статуя бога-рептилоида (крокОдил, что ли =)), вот это вот всё =).

The Invisible Web: What It Is, How You Can Find It

How Big is the Invisible Web?

Why Is It Called "The Invisible Web"?

Why Is The Invisible Web Important?

How Do I Use The Invisible Web?

Humanities

Specific to U.S. Government

Health and Science

Mega-Portals

What About Other Invisible Web Resources?

The Bottom Line About The Invisible Web

Related Articles

Our Expert Recommends

Page Summary

Style Credit

Expand Cut Tags