[extropy-chat] how does google find out everything?

[Eugen Leitl]

On Tue, Aug 22, 2006 at 03:12:14PM -0400, Robert Bradbury wrote:

>      I have a lot of data on my site, so about 90% of my traffic is due
>      to spiders.
> 
>    I believe you can fix that by adjusting your robots.txt file,  e.g.
>    User-agent: *
>    Crawl-Delay: 120

Oh, I don't mind the load, most of it is static files which lighttpd
serves far better than Apache could (yeah, I know about comanche & Co
and what the benchmarks say), despite running in a virtual server,
on a measly 1.2 GHz Athlon XP.

>    I'm currently noticing regular but not excessive crawling by msnbot
>    using that.
>    I'm less sure about Yahoo & Google.

I want the material to be found and indexed, and the load is negligible,
and the traffic is very cheap, so I don't mind the spiders crawling.
As long as there aren't too many of them, so they'd crawl the site daily.

-- 
Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org
______________________________________________________________
ICBM: 48.07100, 11.36820            http://www.ativel.com
8B29F6BE: 099D 78BA 2FD3 B014 B08A  7779 75B0 2443 8B29 F6BE
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 191 bytes
Desc: Digital signature
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20060822/c4c92d12/attachment.bin>