Search Engines :: TheServerPages.com

:: TSP Menu
- Home
- Authors
- Glossary
- RSS Feed
- Contact Us

:: Servers
- CPanel
- Hosting
- Linux
- Windows

:: Webmaster
- Databases
- Design
- PHP
- Search Engines

:: Scripts
- Scripts

:: Documentation
- PHP Manual

>> TheServerPages Articles >> Webmasters >> Search Engines

Articles:

Robots.txt Format by Wojjie (2004-07-19)
When search engines crawl through the internet, they check for the existance of the 'robots.txt' file. This file tells the search engine what urls it should not crawl to, and has a simple file format.

Links: Game-Monitor Home-Recipes Game Server Sites