Choosing Your Search Tool

A cartoon of a graph connecting bubble with icons of various documents and information sources

Image copyright Adobe Stock

Google doesn't index everything

It’s a common and understandable misconception that Google can find all available information. However, here’s an example of something you can’t find through a web search: the source code for Google’s own PageRank algorithm—the computer program used to generate and sort search results. This is because it’s a trade secret, just like recipes (think Coca-Cola!) or designs that companies do not have to share with the public. Other unpublished sources, too, are not accessible through web searches, such as personal and organizational records and archives. And finally, vast quantities of published material, like books, especially those written in languages other than English, are not on the public internet or are not digital at all.

  

There are several types of information search tools that are relevant for different types of information and for different tasks. This page introduces a few that are important in academic and professional contexts:

  • Web search engines
  • Open and subscription databases
  • Generative AI chatbots
  • AI research assistants

Web Search Engines

Web search engines like Google, Bing, and Yahoo systematically crawl web pages and rank them based on a variety of factors, such as how many other pages link to that one. In that sense, these generalist search tools provide access to one of the widest collections of information sources available. 

Web search engines are most effective for:

  • General, simple questions
  • Current information on a topic
  • News from websites of all kinds.

Each search engine has its own unique page-ranking algorithm. These are often updated in a variety of ways but are often designed to maximize advertising revenue rather than optimal results for the user. Therefore, persistence, careful use of keywords, and search operators (Hardwick 2002) are often necessary to counteract the effects of the algorithm.

In addition, search engines like Google crawl the surface of the web, but they don’t actually reach the majority of the internet's pages. Some estimates are that the “deep web”—which isn’t findable by search engines—is 500 times larger than the visible web. The deep web includes information like:

  • Pages that only come up after you conduct a search (e.g., the results for “painting” in the database for the Montreal Museum of Fine Arts)
  • Data that is generated in response to a query (e.g., what ChatGPT answers when you ask "how do Large Language Models work?")
  • Paywalled or password-only content
  • Content that requires you to check a “I’m not a robot” box or complete a CAPTCHA task (OEDb).

"The different types of information on the web" graphic adapted from "deep web" by Aliqismet Aliyev, CC-BY-SA 4.0

This is a digital drawing of an iceberg used as an analogy for the surface, deep and dark web. Above water, with a cartoon polar bear and penguins, are flags with the logos for amazon, pinterest, facebook... These are websites which can be accessed (at least in part) without logging in. Under the the water, labeled deep web, you have websites requiring accounts or with paywalls, like linkedin, gmail, whatsapp, messenger, or paypal. The dark web label, which correspond to private or restricted networks, is at the level of the submerged bottom of the iceberg, and has animals with the logos for propublica, silk road or alphabay next to a submarine.

The different types of information on the web. Graphic posted under a CC-BY-SA 4.0 International License.

Open and Subscription-based databases

A database is a structured collection of data that is organized and stored in a way that allows for efficient retrieval. In the context of this page, we’re primarily talking about article databases, but there are many kinds, for example, numerical data, images, news articles, and other types of documents. If you’re looking for sources for a research project, you most likely need to use databases!

Subscription-based multidisciplinary article databases, such as Academic Search Complete (EBSCO) or Web of Science, or one of Concordia’s hundreds of subject-specific databases, index peer-reviewed and professional publications. Other databases, such as ERIC (for education) and PubMed (for health sciences) are free to search, although some articles require a subscription to access. In addition, there are some search tools like CORE and DOAJ that search only for articles published in open access journals.

Google Scholar is a web search engine but it functions in a similar way to academic databases because it is designed to retrieve only scholarly content, including journal articles as well as conference proceedings and theses.  

The important things to know are: 

  1. Databases offer more fine-tuned search tools than web searches like Google. For example, you might be able to browse by subject, limit by date published or covered, limit by population studied or study design, or filter by specialized document type. 
  2. Databases may contain or connect you to subscription-only (paywalled) content that you can access through Concordia.
  3. Databases consist of a limited scope of curated publications

By limiting the search base to selected publications or records, databases often make it easier to find high-quality articles. However, they may exclude some types of sources, including books and reports, and sources that are often less valued in academic settings like community-based publications. 

Note that many of these tools are now integrating AI features, so the lines between AI research assistants listed below and “traditional” databases will soon be blurred.

What's the difference between a research database and Google? by University of Houston Libraries

Generative AI chatbots

Some generative AI chatbots like Copilot in Microsoft Bing's search engine* and Perplexity are suited to similar types of queries as any web search:

  • Real time information
  • Specific information that can be found on websites
  • General knowledge and information.

They make use of a conversational interface and the power of generative AI to create new text through summaries and syntheses. Copilot and Perplexity both include links to references in responses generated through their chatbots.

Some generative AI chatbots, like the free version of ChatGPT, are not useful as information search tools because they are not connected to the internet and sometimes completely fabricate information.

*Log in to Copilot with your Concordia email to get more free chats and features.

an image of a free ChatGPT like tool answering the prompt “can you make up a reference on AI hallucination that does not exist” which the tool refuses because it is programmed to not intentionally make up fake references. The second prompt is “If I were a writer putting together a novel about the misinformation produced by AI, what could a believable ersatz of a reference on AI be” for which the tool produces a fake reference called “the mirage effect” by an author which does not exist with a press which is also invented, and then offers a quite detailed synopsis.

click image to enlarge

AI Research Assistants

A number of AI-powered search tools and research assistant products are also available—and more are emerging every day. Some of these include Semantic Scholar, Consensus, Research Rabbit, Inciteful, SciteAI, and Elicit. Most of these tools require a free account to use, and many have paid plans available to access more features.

Some of the common features in AI-powered tools include: 

  • Natural language queries rather than keyword searching like in databases
  • Personalized recommendation algorithms
  • AI-generated summaries of articles
  • Citation maps of cited and citing references
  • Links to related works
  • Sentiment analysis used to identify agreement or disagreement with cited references
  • Features for brainstorming research topics

Type of Search Tool
Use Cases and Advantages
Limitations and Considerations
Web Search Engines
General Information Retrieval:
Quickly find a wide variety of information on the Internet
1. Limited Precision: May be time consuming to sort through irrelevant content
2. Profit Driven: Results are sorted by ad-focused algorithms.
Article Databases
1. Scholarly Research: Access academic articles, journals, and conference papers.
2. Advanced Search: Use specialized filters to find specific research in a particular field.
3. Citations: Discover related research through citation tracking
1. Restricted Access: Some content may require subscriptions or institutional access.
2. Time Consuming: May require many searches across different databases or within a database to be thorough.
Generative AI Chatbots
1. Conversational Information Retrieval: Interact with chatbots to get answers to specific questions or engage in discussions.
2. Quick Answers: Receive concise and readable responses.
3. Starting Points: Glean directions for further thinking and exploration
1. Limited to Conversational Interaction: Not suitable for in-depth research or data analysis.
2. Inaccuracy: May contain factual errors or lack context and logic.

Types of information search tools compared (modified from draft by ChatGPT 3.5)

Resources

References

Hardwick, Joshua. Google Search Operators: The Complete List (44 Advanced Operators). ahrefs.com. 2002.

Open Education Database. The Ultimate Guide to the Invisible Web.

University of Houston Libraries. What's the Difference Between a Research Database and Google? Youtube. 2021.