![]() |
|
The format for this standard is very simple. It consists of records in a text file, each record consisting of two fields: a user-agent line and one or more disallow lines. These fields are formatted in a specific way so that the spider program can read them. Business - List of business/finance/loan/mortgage resources Computers - List of computer hardware/software/peripheral resources Internet - List of webhosting/webdesign/internet marketing resources Software - List of software resources Web Design - List of web design/development resources Web Hosting - List of web hosting resources Web Promotion - List of search engine optimization/internet marketing resources Web Resources - List of other web resources Recreation - List of travel/hotel/cruise resources Casino - List of online gambling/poker/blackjack/roulette resources Health - List of online pharmacy/hospital/health resources Shopping - List of online shopping/gift resources Miscellaneous - List of all other resources not stated above
The first field is the "User-agent" field, which his used to specify which robot the "Disallow" lines in the next field apply to. Usually, this contains the wildcard character "*" to specify all robots. In some cases, however, you may wish to only exclude specific robots, such as the googlebot. |
|
| © Copyright 2006, innovationdigital.com |