Sorry, last message was a mistype...<br><br><div><span class="gmail_quote">On 8/22/06, <b class="gmail_sendername">Eugen Leitl</b> <<a href="mailto:eugen@leitl.org">eugen@leitl.org</a>> wrote:</span><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
The major engines (Yahoo, Microsoft, Google) typically hit 1-2 times/month.<br>I have a lot of data on my site, so about 90% of my traffic is due to spiders.</blockquote><div><br>I believe you can fix that by adjusting your
robots.txt file, e.g.<br><br>User-agent: *<br>Crawl-Delay: 120<br><br>I'm currently noticing regular but not excessive crawling by msnbot using that.<br>I'm less sure about Yahoo & Google.<br><br>R.<br> <br></div><br>
</div><br>