If we built a free spider that operated only via the file system, which
would build an index mapped to URL-space, then offered to serve those
indexes from here, would people use it? By not speaking HTTP, this indexer
would be inherently incapable of being used anywhere but locally.
In other words, as was suggested here, you'd maintain your index locally,
then ship it to Verity to be served by our Web server.
Obviously we wouldn't want to burden ourselves with maintaining an index to
the entire net, but we might be willing to do this for free for sites that
are focused on the Web, publishing, libraries, etc. -- especially
educational and non-profit sites. We're building a "virtual library" of
such information here; this would make it more efficient... although I'm
not sure that I want to have to depend on others to keep the indexes up to
date. I suppose we could have a spider that would spot check last-modified
dates to make sure that indexes are current...
Thoughts?
Nick