mirror of
https://github.com/python/cpython.git
synced 2025-11-02 11:08:57 +00:00
Minor changes to match the style guide.
This commit is contained in:
parent
4124a0b343
commit
8ee679f0ce
21 changed files with 79 additions and 72 deletions
|
|
@ -9,13 +9,13 @@
|
|||
\sectionauthor{Skip Montanaro}{skip@mojam.com}
|
||||
|
||||
\index{WWW}
|
||||
\index{World-Wide Web}
|
||||
\index{World Wide Web}
|
||||
\index{URL}
|
||||
\index{robots.txt}
|
||||
|
||||
This module provides a single class, \class{RobotFileParser}, which answers
|
||||
questions about whether or not a particular user agent can fetch a URL on
|
||||
the web site that published the \file{robots.txt} file. For more details on
|
||||
the Web site that published the \file{robots.txt} file. For more details on
|
||||
the structure of \file{robots.txt} files, see
|
||||
\url{http://info.webcrawler.com/mak/projects/robots/norobots.html}.
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue