Get the value of a key from the cache and deserialize it :param key: :return:

GET request with header signing Args: uri: Request route params: Request parameters return_ori_co

GET request with header signing Args: uri: Request URI params: Request parameters Returns:

GET request, signs request headers Args: uri: Request route params: Request parameters Returns:

GET请求

昵称中间脱敏：首尾各保留 1 字，中间替换为星号。 - 长度 <= 1：返回 "*" - 长度 == 2：首字 + "*" - 长度 >= 3：首字 + "***" + 尾字 这样既保留教学分析所需的内容归属语义，又无法据昵称定位到真人。

把原始用户 ID 转成匿名哈希，用于内容/评论记录的创作者分组， 不暴露真实身份。返回 sha256 截断 16 位的十六进制串。

Get all keys matching the pattern First try KEYS command, if not supported fallback to SCAN

Set the value of a key in the cache and serialize it :param key: :param value: :param expire_time: :return:

创建统一配置的 httpx.AsyncClient。 从配置文件读取 DISABLE_SSL_VERIFY（默认 False，即开启 SSL 验证）。 仅在使用企业代理、Burp、mitmproxy 等中间人代理时才需将其设为 True。

Save or update data (upsert)

Get or create a singleton instance for the given platform and crawler type Args: platform: Platform name (xhs, dy, ks, e

Abstract method to get IP, different HTTP proxy providers need to implement this method :param num: Number of IPs to extract

Create log entry

Cleanup resources Args: force: Whether to force cleanup browser process (ignoring AUTO_CLOSE_BROWSER config)

Create IP proxy pool :param ip_pool_count: Number of IPs in the pool :param enable_validate_ip: Whether to enable IP proxy validation

Push log to queue

Close browser context

Check if proxy IP has expired Args: buffer_seconds: Buffer time (seconds), how many seconds ahead to consider expired to

Search for notes and retrieve their comment information. Returns:

Check if proxy is expired and automatically refresh if necessary

Extract text from HTML, removing all tags.

Query a single record

Check if current proxy has expired Args: buffer_seconds: Buffer time (seconds), how many seconds ahead to consider expire

Get browser info

Add cookies

Launch browser and connect via CDP

Fetch current Tieba PC JSON APIs from the browser context. These APIs rely on logged-in browser cookies and Baidu's PC signing

Get current timestamp (13 digits): 1701493264496 :return:

Initialize proxy pool reference Args: proxy_ip_pool: Proxy IP pool instance

Auto-adjust column widths based on content Args: sheet: Worksheet object

Connect to browser via CDP

Write headers to sheet Args: sheet: Worksheet object headers: List of header names

Write data row to sheet Args: sheet: Worksheet object data: Data dictionary headers: List of hea

获取抖音媒体，自动判断媒体类型是短视频还是帖子图片并下载 Args: aweme_item (Dict): 抖音作品详情

Get collection: {prefix}_{suffix}

Wrapper for httpx common request method, processes request response Args: method: Request method url: Request

Safely convert a raw config value to an enum member.

extract zhihu author Args: author: Returns:

把 KuaishouStoreFactory.create_store 替换为返回捕获用 FakeStore 的 staticmethod。 FakeStore 把 store_content/store_comment 收到的 dict 原样写入 holder.content / hold

把 DouyinStoreFactory.create_store 替换为返回 fake,返回原方法用于还原。

Create cache object :param cache_type: Cache type :param args: Arguments :param kwargs: Keyword arguments :re

Extract data-field value from selector Args: selector: Selector object Returns: Dictionary containin

Get post details by post ID (uses Playwright to access page, avoiding API detection) Args: note_id: Post ID Retu

Get the information and comments of the specified post Args: note_id_list: Returns:

Parse creator ID from Bilibili creator space URL Args: url: Bilibili creator space link - https://space.bilibili.com/4343

Common request method wrapper for requests, handles request responses Args: method: Request method url: Reque

Build main.py command line arguments

Establish connection

把 store.weibo.WeibostoreFactory.create_store 替换为返回 fake 的静态方法, 返回 (module, orig) 便于 finally 还原。

请求头参数签名 (使用 xhshow 纯算法) Args: url: 请求 URI path params: GET 请求参数 payload: POST 请求参数 Returns:

Batch get content comments Args: content_list: Returns:

Batch get note comments

Batch get note comments

batch get notes comments :param note_id_list: :return:

batch get video comments :param video_id_list: :return:

batch get video comments :param video_id_list: :return:

Close connection

extract zhihu comments Args: page_content: zhihu content object comments: zhihu comments Returns: