While checking the logs of one of my websites I noticed something rather weird.
Some person (22.214.171.124) with User-Agent/browser “Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; ……/1.0 )” was downloading _all_ files from my website. Totally ignoring robots.txt and requesting pages without providing a referral.
This seemed quite odd and didn’t seem to be a decent/real search robot. It kept requesting files every 3-4 seconds for about one hour. Decent search bots try to spread the load over a few minutes, and wouldn’t request about 1000 files (1.6 Gb) at once.
So, well, I banned it.