User-Agent ……/1.0

While checking the logs of one of my websites I noticed something rather weird.

Some person (65.210.123.237) with User-Agent/browser “Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; ……/1.0 )” was downloading _all_ files from my website. Totally ignoring robots.txt and requesting pages without providing a referral.

This seemed quite odd and didn’t seem to be a decent/real search robot. It kept requesting files every 3-4 seconds for about one hour. Decent search bots try to spread the load over a few minutes, and wouldn’t request about 1000 files (1.6 Gb) at once.

After some Googling I noticed I wasn’t the first person wondering what it was. And as it seems, no one had a clear answer who this bot is, or what it purpose is.

So, well, I banned it.

Leave a Reply...