mirror of
https://github.com/python/cpython.git
synced 2025-11-02 03:01:58 +00:00
bpo-21475: Support the Sitemap extension in robotparser (GH-6883)
This commit is contained in:
parent
7a1c027501
commit
5db5c0669e
5 changed files with 47 additions and 0 deletions
|
|
@ -76,6 +76,15 @@ structure of :file:`robots.txt` files, see http://www.robotstxt.org/orig.html.
|
|||
|
||||
.. versionadded:: 3.6
|
||||
|
||||
.. method:: site_maps()
|
||||
|
||||
Returns the contents of the ``Sitemap`` parameter from
|
||||
``robots.txt`` in the form of a :func:`list`. If there is no such
|
||||
parameter or the ``robots.txt`` entry for this parameter has
|
||||
invalid syntax, return ``None``.
|
||||
|
||||
.. versionadded:: 3.8
|
||||
|
||||
|
||||
The following example demonstrates basic use of the :class:`RobotFileParser`
|
||||
class::
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue