Getting Into Google

[The Bar Keep]

Dan started out his presentation discussing what “indexing” means and how Google goes about it. Basically, the process for the Google crawler is to first look at the robots.txt file in order to learn where it shouldn’t go, and then it gets down to business visiting the pages it is allowed to visit. As the crawler lands on a page, it finds the relevant information contained on it, then follows each link and repeats the process.

Full article: http://www.highrankings.c...isor/getting-into-google/