mirrors/cpython
1
0
Fork 0
mirror of https://github.com/python/cpython.git synced 2025-08-03 08:34:29 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 6

Merge Issue #22558.

Browse source
This commit is contained in:
Terry Jan Reedy 2016-06-11 15:06:08 -04:00
commit 4da945f361
224 changed files with 542 additions and 136 deletions
Download patch file Download diff file Expand all files Collapse all files

4
Doc/library/urllib.robotparser.rst
View file

@ -4,8 +4,10 @@
.. module:: urllib.robotparser
:synopsis: Load a robots.txt file and answer questions about
fetchability of other URLs.
.. sectionauthor:: Skip Montanaro <skip@pobox.com>
**Source code:** :source:`Lib/urllib/robotparser.py`
.. index::
single: WWW
@ -13,6 +15,8 @@
single: URL
single: robots.txt
--------------
This module provides a single class, :class:`RobotFileParser`, which answers
questions about whether or not a particular user agent can fetch a URL on the
Web site that published the :file:`robots.txt` file. For more details on the