[extropy-chat] Re: Identity Erasure? or faulty platforms?

[Amara Graps]

On Sat, 13 Nov 2004 16:13:07 +0100, Patrick Wilken
<patrick.wilken at nat.uni-magdeburg.de> wrote:
> 
> On 13 Nov 2004, at 16:08, Amara Graps wrote:
> > True, they should. I am sufficiently annoyed with companies that don't
> > adhere by netiquette's courtesies (i.e. respecting the wishes of the
> > web
> > site's robot.txt file), that I usually block their future access to my
> > web site (using .htaccess) if I think that they've behaved badly.
> 
> Amara:
> 
> How often does this happen?

I don't know the true frequency because I check my logs randomly,
and I don't always check if the cases when a person downloaded my web
site (~1000 files!) whether there was a check on the robot.txt file
first. Both situations will cause me to bar future entry to my web site
via .htaccess (the reason for the second, is that I think it's rude, the
reason for the first is that I think that they are spammers, and
spammers have caused me no end of grief these last years)

So my best guess for how often a robot ignores my robots.txt file is a
couple of times per month.

Amara