Re: Byte ranges (actually robots)

Gavin Nicol ([email protected])
Thu, 25 May 1995 09:59:25 +0500


>So what to do? If we recognize the user-agent as a known robot, we give it
>a flat presentation of the structure. The catch of course, is that there
>is no standard way to recognize a robot. Some convention in the UserAgent
>field would be sufficient, but....

Isn't there a standard permissions file that robots can fetch?