While checking the logs of one of my websites I noticed something rather weird.
Some person (65.210.123.237) with User-Agent/browser “Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; ……/1.0 )” was downloading _all_ files from my website. Totally ignoring robots.txt and requesting pages without providing a referral.
This seemed quite odd and didn’t seem to be a decent/real search robot. It kept requesting files every 3-4 seconds for about one hour. Decent search bots try to spread the load over a few minutes, and wouldn’t request about 1000 files (1.6 Gb) at once.
After some Googling I noticed I wasn’t the first person wondering what it was. And as it seems, no one had a clear answer who this bot is, or what it purpose is.
So, well, I banned it.