Re: What if we offered a local spider?

Nick Arnett ([email protected])
Mon, 17 Oct 1994 09:56:07 -0500


Brian Behlendorf wrote:

> Right, it sounds like this is WAIS++, inverted. Not only should the
> index be generated locally, the engines should run locally as well. The
> Internet just doesn't like centralized anything. The only benefit of
> having multiple sites' indices on one machine would be search queries
> that return hits from more than one site - WebCrawler, in other words.
> What's needed is some way to distribute this search, so that search
> engines on all the different machines could return results with the user
> just asking once.

The distributed approach requires much more client intelligence to
distribute the query across servers, gather up results and relevancy-rank
them. Sounds like Z39.50 to me; shall we just add it to the clients? ;-)

The more likely scenario is that Z39.50 will be added to servers, so that
one server can integrate a number of others. It's just not going to happen
in the clients any time soon.

Nick