mirrors/cpython
1
0
Fork 0
mirror of https://github.com/python/cpython.git synced 2025-07-19 01:05:26 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 8

SF patch #911431: robot.txt must be robots.txt

Browse source 
(Contributed by George Yoshida.)
This commit is contained in:
Raymond Hettinger 2004-03-13 20:27:23 +00:00
parent 3aa82c07f7
commit 2d95f1ad57
2 changed files with 3 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

4
Lib/robotparser.py
View file

@ -83,7 +83,7 @@ class RobotFileParser:
self.entries.append(entry)
def parse(self, lines):
"""parse the input lines from a robot.txt file.
"""parse the input lines from a robots.txt file.
We allow that a user-agent: line is not preceded by
one or more blank lines."""
state = 0
@ -148,7 +148,7 @@ class RobotFileParser:
def can_fetch(self, useragent, url):
"""using the parsed robots.txt decide if useragent can fetch url"""
_debug("Checking robot.txt allowance for:\n user agent: %s\n url: %s" %
_debug("Checking robots.txt allowance for:\n user agent: %s\n url: %s" %
(useragent, url))
if self.disallow_all:
return False

2
Misc/cheatsheet
View file

@ -1962,7 +1962,7 @@ repr Redo repr() but with limits on most sizes.
rexec Restricted execution facilities ("safe" exec, eval, etc).
rfc822 RFC-822 message manipulation class.
rlcompleter Word completion for GNU readline 2.0.
robotparser Parse robot.txt files, useful for web spiders.
robotparser Parse robots.txt files, useful for web spiders.
sched A generally useful event scheduler class.
sets Module for a set datatype.
sgmllib A parser for SGML.