hub / github.com/Tjatse/node-readability

github.com/Tjatse/node-readability @v0.4.5

Chat with this repo

repository ↗ · DeepWiki ↗ · release v0.4.5 ↗ · + Follow

16 symbols 46 edges 18 files 15 documented · 94% 1 cross-repo links

What it actually does AI analysis from the code graph — generated when you open this

loading…

README

read-art

Readability reference to Arc90's.
Scrape article from any page (automatically).
Make any web page readable, no matter Chinese or English.

快速抓取网页文章标题和内容，适合node.js爬虫使用，服务于ElasticSearch。

Features

Fast And Shoot Straight.
High Performance - Less memory
Automatic Read Title & Content
Follow Redirects
Automatic Decoding Content Encodings(Avoid Messy Codes, Especially Chinese)
Gzip/Deflate Support
Proxy Support
Auto-generate User-Agent
Free and extensible

Performance

In my case, the speed of spider is about 700 thousands documents per day, 22 million per month, and the maximize crawling speed is 450 per minute, avg 80 per minute, the memory cost are about 200 megabytes on each spider kernel, and the accuracy is about 90%, the rest 10% can be fixed by customizing Score Rules or Selectors. it's better than any other readability modules.

Server infos: * 20M bandwidth of fibre-optical * 8 Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz cpus * 32G memory

Installation

npm install read-art --production

Usage

read(html/uri [, options], callback)

It supports the definitions such as: * html/uri Html or Uri string. * options An optional options object, including: - output The data type of article content, head over to Output to get more information. - killBreaks A value indicating whether or not kill breaks, blanks, tab symbols(\r\t\n) into one `

,trueby default. - **minTextLength** If the content is less than[minTextLength]characters, don't even count it,25by default. - **minParagraphs** A number indicates whether or not take the top candidate as a article candidate,3by default, i.e.: IftopCandidatedom has more than3

children,topCandidatewill be considered as the article dom, otherwise, it will be the parent oftopCandidate(not). - **tidyAttrs** Remove all the attributes on elements,falseby default. - **dom** Will return the whole cheerio dom when this property is set totrue,falseby default, try to useart.domto get the dom object in callback function. - **damping** The damping to calculate score of parent node,1/2by default. e.g.: the score of current document node is20, the score of parent will be20 * damping. - **scoreRule** Customize the score rules of each node, one arguments will be passed into the callback function, [read more](#score_rule). - **selectors** Customize the data extract [selectors](#selectors). - **imgFallback** Customize the way to get source of image, [read more](#imgfallback). - **thresholdScore** A number/function indicates whether or not drop the article content, [read more](#threshold_score). - **thresholdLinkDensity** A0~1decimal indicates whether or not drop the article content, [read more](#threshold_linkdensity). - **options from [cheerio](https://github.com/cheeriojs/cheerio)** - **options from [req-fast](https://github.com/Tjatse/req-fast)** * **callback** Fire after the article has been crawled -callback(error, article, options, response), arguments are: - **error**Errorobject when exception has been caught. - **article** The article object, including:article.title,article.contentandarticle.html. - **options** The request options. - **response** The response of your request, including:response.headers,response.redirects,response.cookiesandresponse.statusCode`.

Head over to test or examples directory for a complete example.

Examples

var read = require('read-art');
// read from google:
read('http://google.com', function(err, art, options, resp){
    if(err){
      throw err;
    }
    var title = art.title,      // title of article
        content = art.content,  // content of article
        html = art.html;        // whole original innerHTML

    console.log('[STATUS CODE]', resp && resp.statusCode);
});
// or:
read({
    uri: 'http://google.com',
    charset: 'utf8'
  }, function(err, art, options, resp){

});
// what about html?
read('<title>node-art</title><body>



hello, read-art!



</body>', function(err, art, options, resp){

});
// of course could be
read({
    uri: '<title>node-art</title><body>



hello, read-art!



</body>'
  }, function(err, art, options, resp){

});

CAUTION: Title must be wrapped in a <title> tag and content must be wrapped in a <body> tag.

With High Availability: spider2

Score Rule

In some situations, we need to customize score rules to crawl the correct content of article, such as BBS and QA forums. There are two effective ways to do this: - minTextLength It's useful to get rid of useless elements (P / DIV), e.g. minTextLength: 100 will dump all the blocks that node.text().length is less than 100.

scoreRule You can customize the score rules manually, e.g.: javascript: scoreRule: function(node){ if (node.hasClass('w740')) { return 100; } }

The elements which have the w740 className will get 100 bonus points, that will make the node to be the topCandidate, which means it's enough to make the text of DIV/P.w740 to be the content of current article.

node The cheerio object.

Example

read('http://club.autohome.com.cn/bbs/thread-c-66-37239726-1.html', {
  minTextLength: 0,
  scoreRule: function(node){
    if (node.hasClass('w740')) {
      return 100;
    }
  }
}, function(err, art){

});

Extract Selectors

Some times we wanna extract article somehow, e.g.: pick the text of .article>h3 as title, and pick .article>.author as the author data:

Example

read({
  html: '<title>read-art</title><body>

<h3 title="--read-art--">Who Am I</h3>

hi, dude, i am <b>readability</b>



aka read-art...

<small class="author" data-author="Tjatse X">Tjatse</small>

</body>',
  selectors: {
    title: {
      selector: '.article>h3',
      extract: ['text', 'title']
    },
    content: '.article p.section1',
    author: {
      selector: '.article>small.author',
      extract: {
        shot_name: 'text',
        full_name: 'data-author'
      }
    }
  },
}, function (err, art) {
  // art.title === {text: 'Who Am I', title: '--read-art--'}
  // art.content === 'hi, dude, i am <b>readability</b>'
  // art.author === {shot_name: 'Tjatse', full_name: 'Tjatse X'}
});

Properties: - selector the query selector, e.g.: #article>.title, .articles:nth-child(3) - extract the data that you wanna extract, could be String, Array or Object.

Notes The binding data will be an object or array (object per item) if the extract option is an array object, title and content will override the default extracting methods, and the output of content depends on the output option.

Image Fallback

Should be one of following types: - Boolean Fallback to img.src = (node.data('src') || node.attr('data-src')) when set to true. - String Customize the attribute name, it will take node.attr([imgFallback]) as src of img. - Function Give users maximum customizability and scalability of source attribute on img, e.g.:

javascript imgFallback: function(node){ return node.attr('base') + '/' + node.attr('rel-path'); }

Example

read({
  imgFallback: true
}, function(err, art){});

read({
  imgFallback: 'the-src-attr'
}, function(err, art){});

read({
  imgFallback: function(node){
    return 'http://img-serv/' + node.attr('relative-path');
  }
}, function(err, art){});

Threshold

Customize the threshold of anchors and nodes' scores.

Score

The thresholdScore is a threshold number which to identify whether or not to discard children of top candidate directly (skip deeper tag/text/link density checking), should be one of following types: - Number A finite number. - Function Calculate the threshold score by yourself, two arguments are passing in: - node The top candidate (mostly like article dom). - scoreKey The data key to storage score, you can get score by node.data(scoreKey).

After read-art got the top candidate, it starts to analyze the children of top candidate, if the score of current child is greater than thresholdScore, the child will be appended to article body directly.

Math.max(10, topCandidate.data(scoreKey) * 0.2) by default.

Example

read({
  thresholdScore: 20
}, function(err, art){});

read({
  thresholdScore: function(node, scoreKey){
    return Math.max(10, node.data(scoreKey) * 0.2);
  }
}, function(err, art){});

Link Density

thresholdLinkDensity is used to identify whether current child of top candidate is a navigator, ad or relative-list, 0.25 by default, so if the text length of anchors in current child devides by text length of top candidate is greater than thresholdLinkDensity, the child will be discarded.

Example

read({
  thresholdLinkDensity: 0.25
}, function(err, art){});

Customize Settings

We're using different regexps to iterates over elements (cheerio objects), and removing undesirable nodes.

read.use(function(){
  //[usage]
});

The [usage] could be one of following: - this.reset() Reset the settings to default. - this.skipTags([tags], [override]) Remove useless elements by tagName, e.g. this.skipTags('b,span'), if [override] is set to true, skiptags will be "b,span", otherwise it will be appended to the origin, i.e. : aside,footer,label,nav,noscript,script,link,meta,style,select,textarea,iframe,b,span

this.regexps.positive([re], [override]) If positive regexp test id + className of node success, it will be took as a candidate. [re] is a regexp, e.g. /dv101|dv102/ will match the element likes `

...or

this.regexps.negative([re], [override]) If negative regexp test id + className of node success, it will not be took as a candidate. [re] is a regexp, e.g. /dv101|dv102/ will match the element likes `

...or

this.regexps.unlikely([re], [override]) If unlikely regexp test id + className of node success, it probably will not be took as a candidate. [re] is a regexp, e.g. /dv101|dv102/ will match the element likes `

...or

this.regexps.maybe([re], [override]) If maybe regexp test id + className of node success, it probably will be took as a candidate. [re] is a regexp, e.g. /dv101|dv102/ will match the element likes `

...or

this.regexps.div2p([re], [override]) If div2p regexp test id + className of node success, all divs that don't have children block level elements will be turned into p's. [re] is a regexp, e.g. /<(span|label)/ will match the element likes <span>... or <label>..., if [override] is set to true, div2p will be /<(span|label)/i, otherwise it will be appended to the origin, i.e. : /<(a|blockquote|dl|div|img|ol|p|pre|table|ul|span|label)/i
this.regexps.images([re], [override]) If images regexp test src attribute of node success, it will be picked as a normal img, otherwise will be dropped. [re] is a regexp, e.g. /\.(gif|jpe?g|png)$/i will match the image that src likes /path/to/foo.jpg, if [override] is set to true, readart.regexps.images will be replaced by `[r

Core symbols most depended-on inside this repo

Shape

Function 16

Languages

TypeScript100%

Modules by API surface

lib/reader.js12 symbols

lib/article.js2 symbols

index.js2 symbols

Used by 1 indexed graphs manifest dependencies, hub-wide

github.com/davidmerfield/blot

For agents

$ claude mcp add node-readability \
  -- python -m otcore.mcp_server <graph>

⬇ download graph artifact