mirrors/cpython
1
0
Fork 0
mirror of https://github.com/python/cpython.git synced 2025-07-19 09:15:34 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 6

SF patch #911431: robot.txt must be robots.txt

Browse source 
(Contributed by George Yoshida.)
This commit is contained in:
Raymond Hettinger 2004-03-13 20:27:23 +00:00
parent 3aa82c07f7
commit 2d95f1ad57
2 changed files with 3 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

4
Lib/robotparser.py
View file

@ -83,7 +83,7 @@ class RobotFileParser:
self.entries.append(entry) self.entries.append(entry)
def parse(self, lines): def parse(self, lines):
"""parse the input lines from a robot.txt file. """parse the input lines from a robots.txt file.
We allow that a user-agent: line is not preceded by We allow that a user-agent: line is not preceded by
one or more blank lines.""" one or more blank lines."""
state = 0 state = 0
@ -148,7 +148,7 @@ class RobotFileParser:
def can_fetch(self, useragent, url): def can_fetch(self, useragent, url):
"""using the parsed robots.txt decide if useragent can fetch url""" """using the parsed robots.txt decide if useragent can fetch url"""
_debug("Checking robot.txt allowance for:\n user agent: %s\n url: %s" % _debug("Checking robots.txt allowance for:\n user agent: %s\n url: %s" %
(useragent, url)) (useragent, url))
if self.disallow_all: if self.disallow_all:
return False return False

2
Misc/cheatsheet
View file

@ -1962,7 +1962,7 @@ repr Redo repr() but with limits on most sizes.
rexec Restricted execution facilities ("safe" exec, eval, etc). rexec Restricted execution facilities ("safe" exec, eval, etc).
rfc822 RFC-822 message manipulation class. rfc822 RFC-822 message manipulation class.
rlcompleter Word completion for GNU readline 2.0. rlcompleter Word completion for GNU readline 2.0.
robotparser Parse robot.txt files, useful for web spiders. robotparser Parse robots.txt files, useful for web spiders.
sched A generally useful event scheduler class. sched A generally useful event scheduler class.
sets Module for a set datatype. sets Module for a set datatype.
sgmllib A parser for SGML. sgmllib A parser for SGML.