[extropy-chat] how does google find out everything?

[Robert Bradbury]

Sorry, last message was a mistype...

On 8/22/06, Eugen Leitl <eugen at leitl.org> wrote:

> The major engines (Yahoo, Microsoft, Google) typically hit 1-2
> times/month.
> I have a lot of data on my site, so about 90% of my traffic is due to
> spiders.


I believe you can fix that by adjusting your robots.txt file,  e.g.

User-agent: *
Crawl-Delay: 120

I'm currently noticing regular but not excessive crawling by msnbot using
that.
I'm less sure about Yahoo & Google.

R.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20060822/23d91bda/attachment.html>