MCPcopy Index your code
hub / github.com/Python3WebSpider/ProxyPool / process

Method process

proxypool/crawlers/base.py:26–32  ·  view source on GitHub ↗

used for parse html

(self, html, url)

Source from the content-addressed store, hash-verified

24 return
25
26 def process(self, html, url):
27 """
28 used for parse html
29 """
30 for proxy in self.parse(html):
31 logger.info(f'fetched proxy {proxy.string()} from {url}')
32 yield proxy
33
34 def crawl(self):
35 """

Callers 2

crawlMethod · 0.95
crawlMethod · 0.80

Calls 2

stringMethod · 0.80
parseMethod · 0.45

Tested by

no test coverage detected