Re: I, Robot

Martijn Koster ([email protected])
Mon, 11 Dec 1995 19:26:55 -0800


Steve <[email protected]> wrote:

>I may have missed it in the archives, but I didn't see any topics on which
> sections of a document a robot or spider or other Web traveller might
> examine to determine content information.

"Might": every part. In practice, who knows, they all do different things.

>If there isn't a specific indexing method that is standard for all search
>engines/robots/crawlers, shouldn't there be one?

What, and give up competition? :-) But yes, it'd help to have better ways.

Frank Atkinson <[email protected]> replied:

> There is a mailing list at [email protected] (put subscribe in the
> subject to subscribe) examining some of these issues as they relate to bots
> which will go "shopping" on the web.

Don't know if that has a different focus, but there is the robots mailing list
on [email protected]

-- Martijn
__________
Email: [email protected]
WWW: http://info.webcrawler.com/mak/mak.html