SEO for Beginners - Part 4

Robots: Robots, or Bots as they are usually called are scripts which are programmed to “crawl” websites and collect information from the site’s meta description tags. These bots then store the information on their servers and index according to the keywords. Consider this example: If you had a site dealing with flower delivery, and your keywords were related to just “roses” and “lillies” then chances of your site showing up for roses or lillies are much higher than showing up for keywords “florists” Keep in mind that you have different kinds of people using the search engines. Some may type in the word roses to research on that flower, showing up your site for this keyword under this scenario might not be what you intended it to be. The catch phrase here is to target your keywords appropriately.
The bots are smart but not intelligent. You would not want your contactus.html page to show up on the search engine. Or any other private page which only you have access to. Or you would not want a page which is under development to show up. By setting up the robots.txt file you can specify what directories within your root directory to be indexed. This way you can hide your other pages from showing up on the search engine.