http://www.webpronews.com Submitted by David A. Utter on Wed, 11/21/2007
The web crawling Googlebot may find a forgotten line in robots.txt that causes it to de-index a site from the search engine.
Webmasters welcome being dropped out of Google about as much as they enjoy flossing with barbed wire. Making it easier for Google to do that would be anathema to being a webmaster. Why willingly exclude one's site from Google?
That could happen with an unvalidated robots.txt file. Robots.txt allows webmasters to provide standing instructions to visiting spiders, which contributes to having a site indexed faster and more accurately.
Google has been considering new syntax to recognize within robots.txt. The Sebastians-Pamphlets blog said Google confirmed recognizing experimental syntax like Noindex in the robots.txt file.
Click here to read more ...
Author
