Software: Apache/2.0.54 (Unix) mod_perl/1.99_09 Perl/v5.8.0 mod_ssl/2.0.54 OpenSSL/0.9.7l DAV/2 FrontPage/5.0.2.2635 PHP/4.4.0 mod_gzip/2.0.26.1a 

uname -a: Linux snow.he.net 4.4.276-v2-mono-1 #1 SMP Wed Jul 21 11:21:17 PDT 2021 i686 

uid=99(nobody) gid=98(nobody) groups=98(nobody) 

Safe-mode: OFF (not secure)

/usr/doc/python-2.2.3/html/lib/   drwxr-xr-x
Free 318.38 GB of 458.09 GB (69.5%)
                            Encoder    Tools    Proc.    FTP brute    Sec.    SQL    PHP-code    Update    Feedback    Self remove    Logout    

Viewing file:     module-robotparser.html (5.98 KB)      -rw-r--r--

Python Library Reference

Previous: 12.18.1 netrc Objects Up: 12. Internet Data Handling Next: 13. Structured Markup Processing

---

12.19 robotparser -- Parser for robots.txt

This module provides a single class, RobotFileParser, which answers questions about whether or not a particular user agent can fetch a URL on the Web site that published the robots.txt file. For more details on the structure of robots.txt files, see http://info.webcrawler.com/mak/projects/robots/norobots.html.

class RobotFileParser()

This class provides a set of methods to read, parse and answer questions about a single robots.txt file.

set_url(url)

Sets the URL referring to a robots.txt file.

read()

Reads the robots.txt URL and feeds it to the parser.

parse(lines)

Parses the lines argument.

can_fetch(useragent, url)

Returns true if the useragent is allowed to fetch the url according to the rules contained in the parsed robots.txt file.

mtime()

Returns the time the robots.txt file was last fetched. This is useful for long-running web spiders that need to check for new robots.txt files periodically.

modified()

Sets the time the robots.txt file was last fetched to the current time.

The following example demonstrates basic use of the RobotFileParser class.

>>> import robotparser
>>> rp = robotparser.RobotFileParser()
>>> rp.set_url("http://www.musi-cal.com/robots.txt")
>>> rp.read()
>>> rp.can_fetch("*", "http://www.musi-cal.com/cgi-bin/search?city=San+Francisco")
0
>>> rp.can_fetch("*", "http://www.musi-cal.com/")
1

---

Python Library Reference

Previous: 12.18.1 netrc Objects Up: 12. Internet Data Handling Next: 13. Structured Markup Processing

---

Release 2.2.3, documentation updated on 30 May 2003.