The Server Pages

»

TheServerPages Articles

»

Webmasters

»

Search Engines

Search Engine Optimization (SEO)


Articles

Robots.txt Format

by Wojjie (2004-07-19)
When search engines crawl through the internet, they check for the existance of the 'robots.txt' file. This file tells the search engine what urls it should not crawl to, and has a simple file format. 



Copyright © 2004-2015: TheServerPages.com