mirror of
https://github.com/codelucas/newspaper.git
synced 2025-12-23 05:36:50 +00:00
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
https://goo.gl/VX41yK
| newspaper | ||
| tests | ||
| .gitignore | ||
| AUTHORS.md | ||
| CHANGES.txt | ||
| HISTORY.md | ||
| LICENSE | ||
| README.rst | ||
| setup.py | ||
Newspaper: Article scraping & curation
======================================
.. image:: https://badge.fury.io/py/textblob.png
:target: http://badge.fury.io/py/textblob
:alt: Latest version
.. image:: https://pypip.in/d/textblob/badge.png
:target: https://crate.io/packages/textblob/
:alt: Number of PyPI downloads
Homepage: `https://textblob.readthedocs.org/ <https://textblob.readthedocs.org/>`_
Inspired by `requests` for its simplicity and powered by `lxml` for its speed; `Newspaper` is a Python 2 library
for extracting articles from the web and curating text (NLP) for keywords, summaries, authors, etc.
Newspaper utilizes async io and caching for speed. Everything is in unicode :)
There are two API's available. The low level article API, and the newspaper API.
.. code-block:: python
>>> import newspaper
>>> cnn_paper = newspaper.build('http://cnn.com')
>>> for article in cnn_paper.articles:
>>> print article.url
u'http://www.cnn.com/2013/11/27/justice/tucson-arizona-captive-girls/'
...
>>> print cnn_paper.category_urls
[u'http://lifestyle.cnn.com', u'http://cnn.com/world', u'http://tech.cnn.com' ...]
>>> print cnn_paper.feeds_urls
[u'http://rss.cnn.com/rss/cnn_crime.rss', ... ]
# Download CNN's listing today:
>>> cnn_paper.download() # IO Heavy, using async requests
>>> print cnn_paper.articles[0].title
u'Police: 3 sisters imprisoned in Tucson home, tortured with music'
>>> print cnn_paper.articles[0].html
u'<!DOCTYPE HTML><html itemscope itemtype="http://...'
>>> print cnn_paper.articles[0].top_img # via reddit's thumbnail alg
u'http://some.cdn.com/3424hfd4565sdfgdg436/
>>> print cnn_paper.articles[0].authors
[u'Eliott C. McLaughlin', u'Some CoAuthor']
>>> print cnn_paper.brand
u'cnn'
>>> for article in cnn_paper.articles:
>>> article.parse() # CPU heavy step, can take 3-6 seconds per
>>> print cnn_paper.articles[0].keywords
[u'music', u'Tucson', ... ]
>>> print cnn_paper.articles[0].text
u'Three sisters who were imprisoned for possibly ... a constant barrage ...'
>>> print cnn_paper.articles[0].summary
u'... imprisoned for possibly ... a constant barrage ...'
Alternatively, you may use newspaper's lower level Article API.
.. code-block:: python
from newspaper import Article
>>> article = Article('http://www.cnn.com/2013/11/27/travel/weather-thanksgiving/index.html?hpt=hp_t1')
>>> article.download()
>>> article.parse()
>>> print article.url
u'http://www.cnn.com/2013/11/27/travel/weather-thanksgiving/index.html'
>>> print article.summary
u'...and so that is how a great Thanksgiving meal is cooked...'
>>> print article.text
u'The purpose of this article is to introduce to you all how to...'
>>> print article.authors
[u'Martha Stewart', u'Bob Smith']
Newspaper stands on the giant shoulders of `lxml`_, `nltk`_, and `requests`_.
.. _`lxml`: https://textblob.readthedocs.org/en/latest/quickstart.html#quickstart
.. _`nltk`: https://textblob.readthedocs.org/en/latest/quickstart.html#quickstart
.. _`requests`: https://textblob.readthedocs.org/en/latest/quickstart.html#quickstart
Features
--------
- Noun phrase extraction
- Part-of-speech tagging
- Sentiment analysis
- Classification (Naive Bayes, Decision Tree)
- Language translation and detection powered by Google Translate
- Tokenization (splitting text into words and sentences)
- Word and phrase frequencies
- Parsing
- `n`-grams
- Word inflection (pluralization and singularization) and lemmatization
- Spelling correction
- JSON serialization
- Add new models or languages through extensions
- WordNet integration
Get it now
----------
::
$ pip install newspaper
Examples
--------
See more examples at the `Quickstart guide`_.
.. _`Quickstart guide`: https://newspaper.readthedocs.org/en/latest/quickstart.html#quickstart
Documentation
-------------
Full documentation is available at https://newspaper.readthedocs.org/.
Requirements
------------
- Python >= 2.6 and <= 2.7*
License
-------
MIT licensed. See the bundled `LICENSE <https://github.com/sloria/TextBlob/blob/master/LICENSE>`_ file for more details.