Software: Apache/2.0.54 (Unix) mod_perl/1.99_09 Perl/v5.8.0 mod_ssl/2.0.54 OpenSSL/0.9.7l DAV/2 FrontPage/5.0.2.2635 PHP/4.4.0 mod_gzip/2.0.26.1a 

uname -a: Linux snow.he.net 4.4.276-v2-mono-1 #1 SMP Wed Jul 21 11:21:17 PDT 2021 i686 

uid=99(nobody) gid=98(nobody) groups=98(nobody) 

Safe-mode: OFF (not secure)

/opt/doc/python-2.3.4/html/Python-Docs-2.3.4/lib/   drwxr-xr-x
Free 318.34 GB of 458.09 GB (69.49%)
                            Encoder    Tools    Proc.    FTP brute    Sec.    SQL    PHP-code    Update    Feedback    Self remove    Logout    

Viewing file:     module-robotparser.html (8.24 KB)      -rw-r--r--

Python Library Reference

Previous: 12.18.1 netrc Objects Up: 12. Internet Data Handling Next: 12.20 csv

---

12.19 robotparser -- Parser for robots.txt

This module provides a single class, RobotFileParser, which answers questions about whether or not a particular user agent can fetch a URL on the Web site that published the robots.txt file. For more details on the structure of robots.txt files, see http://www.robotstxt.org/wc/norobots.html.

class RobotFileParser(

)

This class provides a set of methods to read, parse and answer questions about a single robots.txt file.

set_url(

url)

Sets the URL referring to a robots.txt file.

read(

)

Reads the robots.txt URL and feeds it to the parser.

parse(

lines)

Parses the lines argument.

can_fetch(

useragent, url)

Returns True if the useragent is allowed to fetch the url according to the rules contained in the parsed robots.txt file.

mtime(

)

Returns the time the robots.txt file was last fetched. This is useful for long-running web spiders that need to check for new robots.txt files periodically.

modified(

)

Sets the time the robots.txt file was last fetched to the current time.

The following example demonstrates basic use of the RobotFileParser class.

>>> import robotparser
>>> rp = robotparser.RobotFileParser()
>>> rp.set_url("http://www.musi-cal.com/robots.txt")
>>> rp.read()
>>> rp.can_fetch("*", "http://www.musi-cal.com/cgi-bin/search?city=San+Francisco")
False
>>> rp.can_fetch("*", "http://www.musi-cal.com/")
True

---

Python Library Reference

Previous: 12.18.1 netrc Objects Up: 12. Internet Data Handling Next: 12.20 csv

---

Release 2.3.4, documentation updated on May 20, 2004.