Yes, but that just let's me tell them what to stay away from. The issue
here is not that they should stay away, but that I should recognize them.
The permissions file is not fetched everytime they come around, and the
spacing between visits tends to be long (to reduce load on any one system),
and I can't just flag anyone who fetches robot.txt as a robot, since of
course a person may fetch it too.
Kee Hinckley Utopia Inc. - Cyberspace Architects=81 617/721-6100
[email protected] http://www.utopia.com/
I'm not sure which upsets me more: that people are so unwilling to accept
responsibility for their own actions, or that they are so eager to regulate
everyone else's.