How Search Engines Work – So You Can Work With Them

So, we know that we need to optimize our sites and pages for the search engines, but do we even have an idea of how the search engines gather the information and work? How do they do what they do? Here is a summary of the basic functions of search engines.

To begin, the search engines use automated programs, called “Spiders” or “Bots.” These programs use the hyperlink structure of the internet to crawl the pages and other documents that make up all that we see (and don’t see) on the Internet.

When a page has been crawled, then the contents are “Indexed” and stored in huge databases. The index needs to be watched closely and managed in an extraordinary way so that when a search is conducted, the billions of documents are sorted and the right information pops up.

As a request or search comes in, the engine pulls from the index all of the document that matches the search request. A basic request will pull information for the word, words, and phrase input by the user. Meaning that the individual words are searched first, then the loose phrase. Only if the user inputs the words or phrase in quotation marks will the engine pull the exact phrase with the words in order for display in the results. The search engines perform this function hundreds of millions of times each day.

But wait, how do the results show up in the order they do on the results page? Well, when the search engine is done with the particular search and has determined which sites and pages are a match, the algorithm (or mathematical equation that is commonly used for sorting) performs a calculation on each of the matching results and determines which is most relevant. These results are then sorted on the page in order of most relevant to least relevant and displayed for the user to see. From here, it is up to the user to decide and choose which one most closely matches what they were looking for.

While this process is not really a long one and can return a result from the search engines almost instantly, these systems, such as Google, Bing, and Yahoo! really are the most complex and process-intensive computers in the world. They manage the calculations (millions of them) within seconds and then process the results, sending them out to the users in an extremely quick and effective way.

In the next article, we will talk about the items on your site that can make it difficult for your page to be indexed and how to help the spiders crawl your site easily.