Called from Crawler.crawl() when the link is crawled. When source=None, the link is not a web page (and was not parsed), or possibly a URLTimeout occured (content size too big).
(self, link, source=None)
| 3247 | return 0.80 |
| 3248 | |
| 3249 | def visit(self, link, source=None): |
| 3250 | """ Called from Crawler.crawl() when the link is crawled. |
| 3251 | When source=None, the link is not a web page (and was not parsed), |
| 3252 | or possibly a URLTimeout occured (content size too big). |
| 3253 | """ |
| 3254 | pass |
| 3255 | |
| 3256 | def fail(self, link): |
| 3257 | """ Called from Crawler.crawl() for link whose MIME-type could not be determined, |