Code
Hub
Workspaces
Following
Trending
Connect
MCP
copy
Create free account
hub
/
github.com/dengqiangxi/zhihu_spider
/ functions
Functions
28 in github.com/dengqiangxi/zhihu_spider
⨍
Functions
28
◇
Types & classes
18
↓ 6 callers
Function
get_data
(table_name, distinct_key)
zhihu_spider/misc/db_tools.py:13
↓ 5 callers
Function
parse_sub_item
(sub_item_obj: dict, sub_item: Item)
zhihu_spider/spiders/ZhihuSpider.py:21
↓ 3 callers
Method
close
Overwrite the close() method of pymysql.connections.Connection With pool, put connection back to pool; Without pool, send the
zhihu_spider/misc/mysql_pool.py:55
↓ 3 callers
Method
put_connection
(self, conn)
zhihu_spider/misc/mysql_pool.py:127
↓ 1 callers
Method
_recreate
(self, *args, **kwargs)
zhihu_spider/misc/mysql_pool.py:50
↓ 1 callers
Function
config_logger
()
zhihu_spider/misc/tools.py:7
↓ 1 callers
Method
get_connection
timeout: timeout of get a connection from pool, should be a int(0 means return or raise immediately) retry_num: how many times will r
zhihu_spider/misc/mysql_pool.py:105
↓ 1 callers
Function
get_ua_list
获取ua列表
zhihu_spider/misc/tools.py:64
↓ 1 callers
Function
hump2underline
驼峰形式字符串转成下划线形式 :param hunp_str: 驼峰形式字符串 :return: 字母全小写的下划线形式字符串
zhihu_spider/misc/tools.py:41
↓ 1 callers
Method
make_requests_from_url
(self, url)
zhihu_spider/spiders/ZhihuSpider.py:50
↓ 1 callers
Method
save_item
(self, item, table_name)
zhihu_spider/pipelines.py:38
↓ 1 callers
Method
size
(self)
zhihu_spider/misc/mysql_pool.py:137
↓ 1 callers
Function
spelling_insert_sql
拼接插入sql
zhihu_spider/misc/tools.py:54
Method
__exit__
Overwrite the __exit__() method of pymysql.connections.Connection Base action: on successful exit, commit. On exception, rollback
zhihu_spider/misc/mysql_pool.py:29
Method
__init__
(self)
zhihu_spider/pipelines.py:29
Method
__init__
(self, **kwargs)
zhihu_spider/spiders/ZhihuSpider.py:41
Method
__init__
(self, *args, **kwargs)
zhihu_spider/misc/mysql_pool.py:24
Method
__init__
(self, size=5, name=None, *args, **kwargs)
zhihu_spider/misc/mysql_pool.py:96
Method
close_spider
(self, spider)
zhihu_spider/pipelines.py:52
Method
execute_query
A wrapped method of pymysql's execute() or executemany(). dictcursor: whether want use the dict cursor(cursor's default type is tuple
zhihu_spider/misc/mysql_pool.py:66
Function
format_avatar
(avatar_url: str)
zhihu_spider/misc/tools.py:35
Function
init_logger
初始化logger :param logger_name: logger_name :param logger_path: logger文件路径 :param debug: 是否debug模式
zhihu_spider/misc/tools.py:12
Method
parse
解析单个用户的详细信息 :param response:
zhihu_spider/spiders/ZhihuSpider.py:53
Method
parser_follow_json
从粉丝和关注者的接口中抽出用户的token
zhihu_spider/spiders/ZhihuSpider.py:155
Method
process_item
(self, item, spider)
zhihu_spider/pipelines.py:33
Method
process_request
(self, request: Request, spider)
zhihu_spider/middlewares.py:34
Method
process_response
(self, request, response, spider)
zhihu_spider/middlewares.py:17
Method
start_requests
(self)
zhihu_spider/spiders/ZhihuSpider.py:46