<9508140153.AA01581@cs> from "Paul Burchard" at Aug 13, 95 09:53:31 pm
> What needs to be decided more precisely is (a) the compression
> algorithm to be applied to the logfile, and (b) the format and
> fields of the logfile. Brian Behlendorf suggested a good minimum
> set of log fields: host, timestamp, referer (although some sort of
> "host hiding" should probably be supported for privacy/security
> reasons). Any suggestions for the compression? It would be nice to
> have something that could be used incrementally by the proxy to
> save space.
We've observed 90% compression of log files across several sites
using gzip's Lempel-Ziv (LZ77). Brian's reduced set of headers
produce similar results (though I've yet to sample enough diverse proxies
to know for sure). Since most proxies log the transactions uncompressed,
keeping requests uncompressed until forwarded or some default expiration
seems reasonable storage-wise.
Ideally, it would be nice for servers to be able to request which level
of reporting they want (Level 0 none; Level 1 tabulation; Level 2 summary
(Brian's prop); Level 3 full w/compression).
Jim.