The way I've dealt with the problem is to list only sites that seem to
be public. The list of sites that I have are gotten from public
sources and I try and cull sites with any sort of restriction on them.
>2) Sites should be able to say ``Go away -- I've done it all for you''
>and have the equivalent of ftp:/ls-lR.Z in /
The trouble with ls-lR, is that it doesn't contain keyword and breadth
information. Without such information, an index is very difficult to
deal with.