Robotstxtorgs

robot_detection is a Python library for finding out if a certain HTTP User Agent header is likely to be a bot/spider, rather than an actual human using a browser. It does this with a blacklist of known robots initially seeded from Robotstxt.org's Robot Database, but some entries have been... Platforms: *nix