bpo-21475: Support the Sitemap extension in robotparser (GH-6883)

This commit is contained in:
Christopher Beacham 2018-05-16 07:52:07 -07:00 committed by Ned Deily
parent 7a1c027501
commit 5db5c0669e
5 changed files with 47 additions and 0 deletions

View file

@ -76,6 +76,15 @@ structure of :file:`robots.txt` files, see http://www.robotstxt.org/orig.html.
.. versionadded:: 3.6
.. method:: site_maps()
Returns the contents of the ``Sitemap`` parameter from
``robots.txt`` in the form of a :func:`list`. If there is no such
parameter or the ``robots.txt`` entry for this parameter has
invalid syntax, return ``None``.
.. versionadded:: 3.8
The following example demonstrates basic use of the :class:`RobotFileParser`
class::