MCPcopy Create free account

hub / github.com/dengqiangxi/zhihu_spider / functions

Functions28 in github.com/dengqiangxi/zhihu_spider

↓ 6 callersFunctionget_data
(table_name, distinct_key)
zhihu_spider/misc/db_tools.py:13
↓ 5 callersFunctionparse_sub_item
(sub_item_obj: dict, sub_item: Item)
zhihu_spider/spiders/ZhihuSpider.py:21
↓ 3 callersMethodclose
Overwrite the close() method of pymysql.connections.Connection With pool, put connection back to pool; Without pool, send the
zhihu_spider/misc/mysql_pool.py:55
↓ 3 callersMethodput_connection
(self, conn)
zhihu_spider/misc/mysql_pool.py:127
↓ 1 callersMethod_recreate
(self, *args, **kwargs)
zhihu_spider/misc/mysql_pool.py:50
↓ 1 callersFunctionconfig_logger
()
zhihu_spider/misc/tools.py:7
↓ 1 callersMethodget_connection
timeout: timeout of get a connection from pool, should be a int(0 means return or raise immediately) retry_num: how many times will r
zhihu_spider/misc/mysql_pool.py:105
↓ 1 callersFunctionget_ua_list
获取ua列表
zhihu_spider/misc/tools.py:64
↓ 1 callersFunctionhump2underline
驼峰形式字符串转成下划线形式 :param hunp_str: 驼峰形式字符串 :return: 字母全小写的下划线形式字符串
zhihu_spider/misc/tools.py:41
↓ 1 callersMethodmake_requests_from_url
(self, url)
zhihu_spider/spiders/ZhihuSpider.py:50
↓ 1 callersMethodsave_item
(self, item, table_name)
zhihu_spider/pipelines.py:38
↓ 1 callersMethodsize
(self)
zhihu_spider/misc/mysql_pool.py:137
↓ 1 callersFunctionspelling_insert_sql
拼接插入sql
zhihu_spider/misc/tools.py:54
Method__exit__
Overwrite the __exit__() method of pymysql.connections.Connection Base action: on successful exit, commit. On exception, rollback
zhihu_spider/misc/mysql_pool.py:29
Method__init__
(self)
zhihu_spider/pipelines.py:29
Method__init__
(self, **kwargs)
zhihu_spider/spiders/ZhihuSpider.py:41
Method__init__
(self, *args, **kwargs)
zhihu_spider/misc/mysql_pool.py:24
Method__init__
(self, size=5, name=None, *args, **kwargs)
zhihu_spider/misc/mysql_pool.py:96
Methodclose_spider
(self, spider)
zhihu_spider/pipelines.py:52
Methodexecute_query
A wrapped method of pymysql's execute() or executemany(). dictcursor: whether want use the dict cursor(cursor's default type is tuple
zhihu_spider/misc/mysql_pool.py:66
Functionformat_avatar
(avatar_url: str)
zhihu_spider/misc/tools.py:35
Functioninit_logger
初始化logger :param logger_name: logger_name :param logger_path: logger文件路径 :param debug: 是否debug模式
zhihu_spider/misc/tools.py:12
Methodparse
解析单个用户的详细信息 :param response:
zhihu_spider/spiders/ZhihuSpider.py:53
Methodparser_follow_json
从粉丝和关注者的接口中抽出用户的token
zhihu_spider/spiders/ZhihuSpider.py:155
Methodprocess_item
(self, item, spider)
zhihu_spider/pipelines.py:33
Methodprocess_request
(self, request: Request, spider)
zhihu_spider/middlewares.py:34
Methodprocess_response
(self, request, response, spider)
zhihu_spider/middlewares.py:17
Methodstart_requests
(self)
zhihu_spider/spiders/ZhihuSpider.py:46