The Server Pages


TheServerPages Articles




Search Engines

Search Engine Optimization (SEO)


Robots.txt Format

by Wojjie (2004-07-19)
When search engines crawl through the internet, they check for the existance of the 'robots.txt' file. This file tells the search engine what urls it should not crawl to, and has a simple file format. 

Copyright © 2004-2015: