MCPcopy Create free account

hub / github.com/csuldw/AntSpider / functions

Functions139 in github.com/csuldw/AntSpider

⨍ Functions 139◇ Types & classes 20
↓ 2 callersFunctionsave_proxy
(proxy, expire_time=None)
scrapy/douban/update_proxy.py:47
↓ 1 callersFunctioncheck_ip_valid
(ip)
scrapy/douban/update_proxy.py:34
↓ 1 callersFunctioncheck_ip_valid
(ip)
scrapy/douban/util.py:99
↓ 1 callersMethodget_actor_ids
(self, meta, response)
scrapy/douban/spiders/movie_meta.py:129
↓ 1 callersMethodget_actors
(self, meta, response)
scrapy/douban/spiders/movie_meta.py:123
↓ 1 callersMethodget_alias
(self, meta, response)
scrapy/douban/spiders/movie_meta.py:190
↓ 1 callersMethodget_alt_name
(self, meta, response)
scrapy/douban/spiders/book_meta.py:68
↓ 1 callersMethodget_author
(self, meta, response)
scrapy/douban/spiders/book_meta.py:84
↓ 1 callersMethodget_author_intro
(self, meta, response)
scrapy/douban/spiders/book_meta.py:100
↓ 1 callersMethodget_binding
(self, meta, response)
scrapy/douban/spiders/book_meta.py:156
↓ 1 callersMethodget_biography
(self, meta, response)
scrapy/douban/spiders/person_meta.py:93
↓ 1 callersMethodget_birth
(self, meta, response)
scrapy/douban/spiders/person_meta.py:74
↓ 1 callersMethodget_birthplace
(self, meta, response)
scrapy/douban/spiders/person_meta.py:85
↓ 1 callersMethodget_book_meta
(self, item)
scrapy/douban/pipelines.py:102
↓ 1 callersMethodget_comment
(self, item)
scrapy/douban/pipelines.py:126
↓ 1 callersMethodget_constellatory
(self, meta, response)
scrapy/douban/spiders/person_meta.py:107
↓ 1 callersMethodget_cover
(self, meta, response)
scrapy/douban/spiders/book_meta.py:47
↓ 1 callersMethodget_cover
(self, meta, response)
scrapy/douban/spiders/movie_meta.py:72
↓ 1 callersMethodget_director_ids
(self, meta, response)
scrapy/douban/spiders/movie_meta.py:108
↓ 1 callersMethodget_directors
(self, meta, response)
scrapy/douban/spiders/movie_meta.py:102
↓ 1 callersMethodget_douban_id
(self, meta, response)
scrapy/douban/spiders/book_meta.py:43
↓ 1 callersMethodget_douban_id
(self, subject, response)
scrapy/douban/spiders/movie_subject.py:87
↓ 1 callersMethodget_douban_id
(self, subject, response)
scrapy/douban/spiders/book_subject.py:47
↓ 1 callersMethodget_douban_id
(self, meta, response)
scrapy/douban/spiders/movie_meta.py:50
↓ 1 callersMethodget_genres
(self, meta, response)
scrapy/douban/spiders/movie_meta.py:144
↓ 1 callersMethodget_imdb_id
(self, meta, response)
scrapy/douban/spiders/movie_meta.py:198
↓ 1 callersFunctionget_ip_list
[{'expire_time': '2019-09-02 20:30:25', 'ip': '119.142.205.84', 'port': '4341'}]
scrapy/douban/update_proxy.py:23
↓ 1 callersMethodget_isbn
(self, meta, response)
scrapy/douban/spiders/book_meta.py:164
↓ 1 callersMethodget_languages
(self, meta, response)
scrapy/douban/spiders/movie_meta.py:166
↓ 1 callersMethodget_media_requests
(self, item, info)
scrapy/douban/pipelines.py:251
↓ 1 callersMethodget_movie_meta
(self, item)
scrapy/douban/pipelines.py:47
↓ 1 callersMethodget_name
(self, meta, response)
scrapy/douban/spiders/book_meta.py:61
↓ 1 callersMethodget_name
(self, meta, response)
scrapy/douban/spiders/movie_meta.py:84
↓ 1 callersMethodget_name_en
(self, meta, response)
scrapy/douban/spiders/person_meta.py:121
↓ 1 callersMethodget_name_zh
(self, meta, response)
scrapy/douban/spiders/person_meta.py:114
↓ 1 callersFunctionget_new_ip
(num)
scrapy/douban/update_proxy.py:18
↓ 1 callersFunctionget_new_ip
()
scrapy/douban/util.py:106
↓ 1 callersMethodget_official_site
(self, meta, response)
scrapy/douban/spiders/movie_meta.py:150
↓ 1 callersMethodget_pages
(self, meta, response)
scrapy/douban/spiders/book_meta.py:140
↓ 1 callersMethodget_person_id
(self, meta, response)
scrapy/douban/spiders/person_meta.py:52
↓ 1 callersMethodget_person_meta
(self, item)
scrapy/douban/pipelines.py:75
↓ 1 callersMethodget_price
(self, meta, response)
scrapy/douban/spiders/book_meta.py:148
↓ 1 callersMethodget_profession
(self, meta, response)
scrapy/douban/spiders/person_meta.py:100
↓ 1 callersFunctionget_proxy
()
scrapy/douban/update_proxy.py:68
↓ 1 callersMethodget_proxy
获取代理 :param choice: :param first: 开始爬取的页数 :param end: 结束爬取的后一页 :return:
scrapy/douban/proxylib.py:70
↓ 1 callersMethodget_publish_date
(self, meta, response)
scrapy/douban/spiders/book_meta.py:132
↓ 1 callersMethodget_publisher
(self, meta, response)
scrapy/douban/spiders/book_meta.py:124
↓ 1 callersMethodget_regions
(self, meta, response)
scrapy/douban/spiders/movie_meta.py:158
↓ 1 callersMethodget_release_date
(self, meta, response)
scrapy/douban/spiders/movie_meta.py:174
↓ 1 callersMethodget_runtime
(self, meta, response)
scrapy/douban/spiders/movie_meta.py:183
↓ 1 callersMethodget_score
(self, meta, response)
scrapy/douban/spiders/book_meta.py:172
↓ 1 callersMethodget_score
(self, meta, response)
scrapy/douban/spiders/movie_meta.py:206
↓ 1 callersMethodget_series
(self, meta, response)
scrapy/douban/spiders/book_meta.py:116
↓ 1 callersMethodget_sex
(self, meta, response)
scrapy/douban/spiders/person_meta.py:65
↓ 1 callersMethodget_slug
(self, meta, response)
scrapy/douban/spiders/book_meta.py:57
↓ 1 callersMethodget_slug
(self, meta, response)
scrapy/douban/spiders/movie_meta.py:91
↓ 1 callersMethodget_storyline
(self, meta, response)
scrapy/douban/spiders/movie_meta.py:232
↓ 1 callersMethodget_sub_name
(self, meta, response)
scrapy/douban/spiders/book_meta.py:76
↓ 1 callersMethodget_subject
(self, item)
scrapy/douban/pipelines.py:26
↓ 1 callersMethodget_summary
(self, meta, response)
scrapy/douban/spiders/book_meta.py:91
↓ 1 callersMethodget_tags
(self, meta, response)
scrapy/douban/spiders/book_meta.py:190
↓ 1 callersMethodget_tags
(self, meta, response)
scrapy/douban/spiders/movie_meta.py:220
↓ 1 callersMethodget_translator
(self, meta, response)
scrapy/douban/spiders/book_meta.py:109
↓ 1 callersMethodget_type
(self, meta, response)
scrapy/douban/spiders/movie_meta.py:62
↓ 1 callersMethodget_user_agent
得到随机user-agent :return:
scrapy/douban/proxylib.py:24
↓ 1 callersFunctionget_valid_proxy
()
scrapy/douban/update_proxy.py:79
↓ 1 callersMethodget_votes
(self, meta, response)
scrapy/douban/spiders/book_meta.py:181
↓ 1 callersMethodget_votes
(self, meta, response)
scrapy/douban/spiders/movie_meta.py:213
↓ 1 callersMethodget_year
(self, meta, response)
scrapy/douban/spiders/movie_meta.py:95
↓ 1 callersFunctionis_match_chinese
(s)
scrapy/douban/validator.py:35
↓ 1 callersMethodsave_book_meta
(self, item)
scrapy/douban/pipelines.py:107
↓ 1 callersMethodsave_comment
(self, item)
scrapy/douban/pipelines.py:132
↓ 1 callersMethodsave_movie_meta
(self, item)
scrapy/douban/pipelines.py:52
↓ 1 callersMethodsave_person_meta
(self, item)
scrapy/douban/pipelines.py:80
↓ 1 callersMethodsave_subject
(self, item)
scrapy/douban/pipelines.py:35
↓ 1 callersMethodtest_proxy
测试代理是否能用 :param ip_port: :param choice: :return:
scrapy/douban/proxylib.py:98
↓ 1 callersMethodupdate_book_meta
(self, item)
scrapy/douban/pipelines.py:116
↓ 1 callersMethodupdate_movie_meta
(self, item)
scrapy/douban/pipelines.py:63
↓ 1 callersMethodupdate_person_meta
(self, item)
scrapy/douban/pipelines.py:91
↓ 1 callersFunctionupdate_proxy_as_invalid
(proxy_ip)
scrapy/douban/update_proxy.py:63
↓ 1 callersFunctionupdate_valid_proxy
()
scrapy/douban/update_proxy.py:115
Method__init__
(self, maxdepth, stats=None, verbose_stats=False, prio=1)
scrapy/douban/middlewares.py:120
Method__init__
(self)
scrapy/douban/proxylib.py:20
Method_filter
(request)
scrapy/douban/middlewares.py:135
Method_warn
()
scrapy/douban/pipelines.py:225
Functioncheck_ip_valid
(ip)
scrapy/douban/proxylib.py:150
Methodcookie
(self, request)
scrapy/douban/spiders/movie_subject.py:75
Methodcookie
(self, request)
scrapy/douban/spiders/book_subject.py:35
Methodfile_path
(self, request, response=None, info=None)
scrapy/douban/pipelines.py:223
Methodfrom_crawler
(cls, crawler)
scrapy/douban/middlewares.py:25
Methodfrom_crawler
(cls, crawler)
scrapy/douban/middlewares.py:78
Methodfrom_crawler
(cls, crawler)
scrapy/douban/middlewares.py:127
Methodget_comments
(self, meta, response)
scrapy/douban/spiders/movie_meta.py:226
Functionget_proxy
()
scrapy/douban/util.py:134
Functioninit_proxy
()
scrapy/douban/util.py:111
Methoditem_completed
(self, results, item, info)
scrapy/douban/pipelines.py:255
Functionmatch_date
(s)
scrapy/douban/validator.py:20
Functionmatch_year
(s)
scrapy/douban/validator.py:12
Methodparse
(self, response)
scrapy/douban/spiders/book_meta.py:197
Methodparse
(self, response)
scrapy/douban/spiders/book_comment.py:45
next →1–100 of 139, ranked by callers