Today, search engines like Google, Bing, and many others have become synonyms for the internet. You talk about any question or topic, and Google comes right in front of you, smiling gullibly with the abundance of data. But do we know how these search engines work in the background and how are they able to fetch thousands of results with a blink of your eyes?
You will be surprised to know that more than 50,000 searches alone happen on Google every second. That is a couple of trillion searches every year. Mind boggling data isn’t it! Ok, let’s not
Any search engine works on three basic principles:
- Crawling – Discover data
- Indexing – Store data in database
- Retrieving or Searching – Fetch results from database
Google or any other search engines use a program called “Spider” to crawl different websites. Sometimes, spiders are also known as “bot”. These spiders or bots are a piece of a program, which is built to discover billions and trillions of contents on the internet.
For example, spider goes to one website and crawls its hundreds of pages. Every page may contain links or hyperlinks to internal as well as external site (some other webpage) pages. Spiders try to crawl all those links and will try to repeat the same process time and again.
Spider keeps on discovering new contents until it builds a significant database on a particular topic. When Google’s bots crawl the page, it looks for all sort of information like content, images, pdf, videos and links. You can right click on any website and click “view page source” option. Bots see this source code page.
Now, you might be thinking how often Google crawls any website? Crawling also depends on various factors. For instance, if your website takes a lot of time to load then Google spider may not crawl your web pages more often. And why not, there are so many out there, waiting for their turn.
Another criterion could be the hierarchy of pages on a website. If the overall interlinking and placement of web page are mixed up or complex then bot may even give up on this structure and may not also give it a try ever.
Spiders crawl through tons of websites are place them in its database, stacked together in the respective sections based on nature of data, type, meaning, etc. This is no easy job for a search engine has to separate out Petabyte (250 bytes) or even Exabyte (260 bytes) of data and group together with the context.
Photo credit: Google
You can imagine search database as a library full of books placed according to the author, category, age, and much more. It is to be noted that search engines may only store the substantial portion of a web page instead storing everything.
If you talk about Google then it has different data centres across the world placed strategically to serve you results in less than a second. For example, if you search for anything in the USA on Google then it will search in database most closer to the USA. Right now, I am in India, and I cannot open www.google.com as this belongs to USA and the nearest Google data centre has this address www.google.co.in.
All these data centres are enormous regarding infrastructure and sometimes has the most sophisticated equipment in place.
#3. Retrieving or Searching
You usually see this step daily whenever you search for anything in Google or Bing. You enter search terms like “nearest grocery store” or maybe something more intellectual as “digital marketing course”, and you get most relevant results in close to 0.60 to 0.80 seconds.
The moment you enter your query and hit enter, the respective search engine goes back to its nearest data centre, where everything is indexed, and try to find out best possible fit. The moment it has sufficient data, it returns and produces results page (SERP) with ranking based on relevancy.
In an SEO (Search engine optimization) perspective this is called ranking of results. The top result generally denotes the best possible answer to your search query. But how Google or any other search decides on the best possible results and that too ranked as one, two, three, etc.? In a way, I want to know the secret recipe or formula what a search engine is using for ranking.
Unfortunately, nobody can know the ranking algorithm but Google or any other search engine. Every search engine is using its algorithm to rank data. This algorithm is ever evolving and takes into maybe couple of hundreds factor before deciding on ranking.
Some of the everyday things which Google looks for while searching its database are the density of keywords, synonyms, data interlinking, website authority, backlinks, etc. Nowadays, Google itself is changing its algorithm on a daily basis. There is no way you catch up with Google and manipulate things.
Day by day the search engine results are becoming more relevant to your search query. It is astonishing to see how these search engines work. Google is using AI (Artificial Intelligence) technology to make their algorithm more human. Days are gone when a website used to rank by stuffing keywords in their content.