MCPcopy Create free account

hub / github.com/dataabc/weiboSpider / functions

Functions132 in github.com/dataabc/weiboSpider

⨍ Functions 132◇ Types & classes 28↳ Endpoints 7
↓ 12 callersMethodto_dict
将对象转换为字典
weibo_spider/user.py:27
↓ 9 callersFunctionhandle_garbled
处理乱码
weibo_spider/parser/util.py:78
↓ 9 callersFunctionhandle_html
处理html
weibo_spider/parser/util.py:50
↓ 7 callersMethod_get_filepath
获取结果文件路径
weibo_spider/spider.py:243
↓ 6 callersFunctionstring_to_int
字符串转换为整数
weibo_spider/parser/util.py:142
↓ 5 callersFunction_is_date
判断日期格式是否正确
weibo_spider/config_util.py:12
↓ 4 callersMethoddownload_one_file
下载单个文件(图片/视频)
weibo_spider/downloader/downloader.py:33
↓ 4 callersFunctionhandle_html_async
异步处理html
weibo_spider/parser/util.py:21
↓ 3 callersMethodextract_user_info
提取用户信息
weibo_spider/parser/info_parser.py:17
↓ 3 callersMethodget_long_weibo
获取长原创微博
weibo_spider/parser/comment_parser.py:21
↓ 2 callersMethod_get_user_id
获取用户id，使用者输入的user_id不一定是正确的，可能是个性域名等，需要获取真正的user_id
weibo_spider/parser/index_parser.py:17
↓ 2 callersMethod_info_to_mongodb
将爬取的信息写入MongoDB数据库
weibo_spider/writer/mongo_writer.py:17
↓ 2 callersMethod_mysql_create
创建MySQL数据库或表
weibo_spider/writer/mysql_writer.py:20
↓ 2 callersMethod_mysql_create_table
创建MySQL表
weibo_spider/writer/mysql_writer.py:43
↓ 2 callersMethod_mysql_insert
向MySQL表插入或更新数据
weibo_spider/writer/mysql_writer.py:49
↓ 2 callersMethod_sqlite_create_table
创建sqlite表
weibo_spider/writer/sqlite_writer.py:22
↓ 2 callersMethod_sqlite_insert
向sqlite表插入或更新数据
weibo_spider/writer/sqlite_writer.py:28
↓ 2 callersMethoddownload_files
下载文件(图片/视频)
weibo_spider/downloader/downloader.py:71
↓ 2 callersMethodextract_avatar_album_url
(self)
weibo_spider/parser/photo_parser.py:12
↓ 2 callersMethodextract_pic_urls
(self)
weibo_spider/parser/album_parser.py:11
↓ 2 callersMethodextract_picture_urls
提取微博原始图片url
weibo_spider/parser/page_parser.py:360
↓ 2 callersMethodextract_preview_picture_list
(self)
weibo_spider/parser/mblog_picAll_parser.py:11
↓ 2 callersMethodget_long_retweet
获取长转发微博
weibo_spider/parser/comment_parser.py:44
↓ 2 callersMethodget_one_page
获取第page页的全部微博
weibo_spider/parser/page_parser.py:68
↓ 2 callersMethodget_page_num
获取微博总页数
weibo_spider/parser/index_parser.py:67
↓ 2 callersMethodhandle_download
下载 urls 里所指向的图片或视频文件，使用 w 里的信息来生成文件名
weibo_spider/downloader/downloader.py:29
↓ 2 callersFunctionhash_url
(url)
weibo_spider/parser/util.py:17
↓ 1 callersFunction_get_config
获取config.json数据
weibo_spider/spider.py:382
↓ 1 callersMethod_mysql_create_database
创建MySQL数据库
weibo_spider/writer/mysql_writer.py:28
↓ 1 callersMethod_sqlite_create
创建sqlite数据库或表
weibo_spider/writer/sqlite_writer.py:14
↓ 1 callersMethod_update_json_data
更新要写入json结果文件中的数据，已经存在于json中的信息更新为最新值，不存在的信息添加到data中
weibo_spider/writer/json_writer.py:18
↓ 1 callersMethod_update_json_data
将获取到的微博数据转换为json输出模式一致
weibo_spider/writer/post_writer.py:23
↓ 1 callersFunctionasync_main
(_)
weibo_spider/spider.py:409
↓ 1 callersMethoddownload_user_avatar
下载用户头像
weibo_spider/spider.py:150
↓ 1 callersMethodget_article_url
获取微博头条文章的url
weibo_spider/parser/page_parser.py:164
↓ 1 callersFunctionget_cookie
Get weibo.cn cookie from Chrome browser
weibo_spider/config_util.py:178
↓ 1 callersMethodget_one_user
获取一个用户的微博
weibo_spider/spider.py:331
↓ 1 callersMethodget_one_weibo
获取一条微博的全部信息
weibo_spider/parser/page_parser.py:317
↓ 1 callersMethodget_original_weibo
获取原创微博
weibo_spider/parser/page_parser.py:109
↓ 1 callersMethodget_picture_urls
获取微博原始图片url
weibo_spider/parser/page_parser.py:264
↓ 1 callersMethodget_publish_place
获取微博发布位置
weibo_spider/parser/page_parser.py:174
↓ 1 callersMethodget_publish_time
获取微博发布时间
weibo_spider/parser/page_parser.py:198
↓ 1 callersMethodget_publish_tool
获取微博发布工具
weibo_spider/parser/page_parser.py:229
↓ 1 callersMethodget_retweet
获取转发微博
weibo_spider/parser/page_parser.py:124
↓ 1 callersMethodget_user
获取用户信息、微博数、关注数、粉丝数
weibo_spider/parser/index_parser.py:30
↓ 1 callersMethodget_user_async
获取用户信息、微博数、关注数、粉丝数
weibo_spider/parser/index_parser.py:46
↓ 1 callersMethodget_user_info
获取用户信息
weibo_spider/spider.py:143
↓ 1 callersMethodget_video_page_url
获取微博视频页面的链接
weibo_spider/parser/comment_parser.py:48
↓ 1 callersMethodget_video_url
获取微博视频url
weibo_spider/parser/page_parser.py:290
↓ 1 callersMethodget_weibo_content
获取微博内容
weibo_spider/parser/page_parser.py:152
↓ 1 callersMethodget_weibo_footer
获取微博点赞数、转发数、评论数
weibo_spider/parser/page_parser.py:242
↓ 1 callersMethodget_weibo_info
获取微博信息
weibo_spider/spider.py:161
↓ 1 callersMethodinitialize_info
初始化爬虫信息
weibo_spider/spider.py:264
↓ 1 callersMethodis_original
判断微博是否为原创微博
weibo_spider/parser/page_parser.py:101
↓ 1 callersMethodsend_post_request_with_token
(self, url, data, token, max_retries, backoff_factor)
weibo_spider/writer/post_writer.py:32
↓ 1 callersMethodstart
运行爬虫
weibo_spider/spider.py:358
↓ 1 callersFunctionto_video_download_url
(cookie, video_page_url)
weibo_spider/parser/util.py:121
↓ 1 callersFunctionupdate_cookie_config
Update cookie in config.json
weibo_spider/config_util.py:188
↓ 1 callersMethodwrite_user
将用户信息写入数据库
weibo_spider/spider.py:138
↓ 1 callersMethodwrite_weibo
将爬取到的信息写入文件或数据库
weibo_spider/spider.py:131
Method__del__
(self)
weibo_spider/writer/kafka_writer.py:40
Method__init__
(self)
tests/test_downloader_async.py:11
Method__init__
(self)
weibo_spider/user.py:8
Method__init__
Weibo类初始化
weibo_spider/spider.py:39
Method__init__
(self)
weibo_spider/weibo.py:13
Method__init__
(self, file_dir, file_download_timeout)
weibo_spider/downloader/img_downloader.py:7
Method__init__
(self, file_dir, file_download_timeout)
weibo_spider/downloader/origin_picture_downloader.py:5
Method__init__
(self, file_dir, file_download_timeout)
weibo_spider/downloader/avatar_picture_downloader.py:7
Method__init__
(self, file_dir, file_download_timeout)
weibo_spider/downloader/video_downloader.py:7
Method__init__
(self, file_dir, file_download_timeout)
weibo_spider/downloader/retweet_picture_downloader.py:5
Method__init__
(self, file_dir, file_download_timeout)
weibo_spider/downloader/downloader.py:16
Method__init__
(self, cookie, user_id, selector=None)
weibo_spider/parser/info_parser.py:12
Method__init__
(self, cookie, weibo_id)
weibo_spider/parser/mblog_picAll_parser.py:6
Method__init__
(self, cookie)
weibo_spider/parser/parser.py:2
Method__init__
(self, cookie, user_config, page, filter, selector=None, defer_fetch=False)
weibo_spider/parser/page_parser.py:21
Method__init__
(self, cookie, weibo_id)
weibo_spider/parser/comment_parser.py:16
Method__init__
(self, cookie, user_id)
weibo_spider/parser/photo_parser.py:6
Method__init__
(self, cookie, user_uri, selector=None)
weibo_spider/parser/index_parser.py:11
Method__init__
(self, cookie, album_url)
weibo_spider/parser/album_parser.py:6
Method__init__
(self, file_path, filter)
weibo_spider/writer/csv_writer.py:10
Method__init__
(self, mongo_config)
weibo_spider/writer/mongo_writer.py:11
Method__init__
根据需要，初始化结果路径、初始化表头、初始化数据库等
weibo_spider/writer/writer.py:5
Method__init__
(self, mysql_config)
weibo_spider/writer/mysql_writer.py:11
Method__init__
(self, sqlite_config)
weibo_spider/writer/sqlite_writer.py:11
Method__init__
(self, file_path)
weibo_spider/writer/json_writer.py:12
Method__init__
(self, kafka_config)
weibo_spider/writer/kafka_writer.py:11
Method__init__
(self, post_config)
weibo_spider/writer/post_writer.py:14
Method__init__
(self, file_path, filter)
weibo_spider/writer/txt_writer.py:10
Method__str__
打印微博用户信息
weibo_spider/user.py:31
Method__str__
打印一条微博
weibo_spider/weibo.py:32
Functionadd_user_uri_list
向user_id_list.txt文件添加若干user_uri
weibo_spider/config_util.py:169
Functionbid2mid
convert string bid to string mid
weibo_spider/parser/util.py:94
Functioncheck_cookie
Checks if user is logged in
weibo_spider/config_util.py:206
Functionget_user_config_list
获取文件中的微博id信息
weibo_spider/config_util.py:115
Methodhandle_download
处理下载相关操作
weibo_spider/downloader/img_downloader.py:12
Methodhandle_download
处理下载相关操作
weibo_spider/downloader/avatar_picture_downloader.py:12
Methodhandle_download
处理下载相关操作
weibo_spider/downloader/video_downloader.py:12
Functionmain
(_)
weibo_spider/spider.py:418
Functionmock_request_get_content
(url, headers)
tests/test_parser/util.py:8
Methodrun_test
()
tests/test_downloader_async.py:28
next →1–100 of 132, ranked by callers