Re: searchable index of the web
Ray Stell ([email protected])
Wed, 30 Jun 93 15:59:37 EDT
>
> I have written a perl script that wanders the WWW collecting URLs, keeping
> tracking of where it's been and new hosts that it finds. Eventually,
> after hacking up the code to return some slightly more useful information
> (currently it just returns URLs), I will produce a searchabe index of this.
> There is a complete list of all the sites it has found at
>
> <a href="http://www.mit.edu:8001/afs/sipb/user/mkgray/ht/comprehensive.html">
> A complete list of sites found by the W4 (World Wide Web Wanderer)
> </a>
>
> I'll announce here when we get this index properly running, however it probably
> won't be until sometime in August, as I am going on vacation. Until then...
>
> Matthew Gray
> [email protected]
>
> Visit the SIPB WWW Plexus Server. URL: http://www.mit.edu:8001/
>
This sounds good, better than depending on my memory. Will w4
crack open an log the <Title> or is this directory tree oriented?
If I'm running a server, how will you know to wander to it?
Should there be a web of wanderers (w5?) so w4e in Europe can
feed from w4u in the US, etc? Why is this topic such a quiet one?
======================================================================
Ray Stell [email protected] (703) 231-4109