Tag: www
-
User-Agent ……/1.0
While checking the logs of one of my websites I noticed something rather weird. Some person (65.210.123.237) with User-Agent/browser “Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; ……/1.0 )” was downloading _all_ files from my website. Totally ignoring robots.txt and requesting pages without providing a referral. This seemed quite odd and didn’t seem to be a decent/real…