Functions — github.com/NanmiCoder/MediaCrawler

↓ 3 callersMethodextract_content_list_from_creator

extract content list from creator Args: anwser_list: Returns:

media_platform/zhihu/help.py:354

↓ 3 callersMethodextract_note_detail_from_api

Extract Tieba post detail from current PC page_pc JSON API.

media_platform/tieba/help.py:156

↓ 3 callersMethodextract_search_note_list

Extract Tieba post list from keyword search result pages, still missing reply count and reply page data Args: page_conten

media_platform/tieba/help.py:330

↓ 3 callersMethodextract_tieba_note_parent_comments_from_api

Extract first-level comments from current PC page_pc JSON API.

media_platform/tieba/help.py:189

↓ 3 callersMethodextract_tieba_note_sub_comments

Extract Tieba post second-level comments from sub-comment page Args: page_content: HTML string of page content

media_platform/tieba/help.py:620

↓ 3 callersFunctionextract_url_params_to_dict

Extract URL parameters to dict

tools/crawler_util.py:226

↓ 3 callersMethodflush

Save workbook to file

store/excel_store_base.py:347

↓ 3 callersMethodflush_all

Flush all Excel store instances and save to files Should be called at the end of crawler execution

store/excel_store_base.py:82

↓ 3 callersFunctiongener_random

(random, option)

libs/douyin.js:232

↓ 3 callersMethodget_bilibili_video

download bilibili video :param video_item: :param semaphore: :return:

media_platform/bilibili/core.py:573

↓ 3 callersMethodget_db

Get database

database/mongodb_store_base.py:47

↓ 3 callersMethodget_note_detail_async_task

Get note detail Args: note_id: xsec_source: xsec_token: semaphore: Returns:

media_platform/xhs/core.py:274

↓ 3 callersMethodget_notice_media

(self, note_detail: Dict)

media_platform/xhs/core.py:463

↓ 3 callersMethodget_or_refresh_proxy

Get current proxy, automatically refresh if expired Call this method before each request to ensure proxy is valid Args:

proxy/proxy_ip_pool.py:130

↓ 3 callersMethodget_video_info_task

Get video detail task :param aid: :param bvid: :param semaphore: :return:

media_platform/bilibili/core.py:419

↓ 3 callersMethodis_connected

Check if connected to browser

tools/cdp_browser.py:510

↓ 3 callersMethodload_all_ip

Load all unexpired IP information from Redis :param proxy_brand_name: Proxy provider name :return:

proxy/base_proxy.py:68

↓ 3 callersFunctionparse_cmd

Parse command line arguments using Typer.

cmd_arg/arg.py:154

↓ 3 callersFunctionparse_video_info_from_url

Parse video ID from Bilibili video URL Args: url: Bilibili video link - https://www.bilibili.com/video/BV1dwuKzmE26/?spm_

media_platform/bilibili/help.py:80

↓ 3 callersFunctionparse_video_info_from_url

Parse video ID from Douyin video URL Supports the following formats: 1. Normal video link: https://www.douyin.com/video/75250824445513106

media_platform/douyin/help.py:101

↓ 3 callersMethodpost

POST request, signs request headers Args: uri: Request route data: Request body parameters Returns:

media_platform/xhs/client.py:187

↓ 3 callersMethodrequest

(self, method, url, **kwargs)

media_platform/kuaishou/client.py:66

↓ 3 callersMethodrequest

(self, method, url, **kwargs)

media_platform/douyin/client.py:123

↓ 3 callersMethodsearch

search

base/base_crawler.py:36

↓ 3 callersMethodsearch

search bilibili video

media_platform/bilibili/core.py:136

↓ 3 callersMethodsearch

(self)

media_platform/kuaishou/core.py:130

↓ 3 callersMethodsearch

Search for notes and retrieve their comment information.

media_platform/xhs/core.py:129

↓ 3 callersMethodsearch

(self)

media_platform/douyin/core.py:127

↓ 3 callersMethodset_ip

Set IP with expiration time, Redis is responsible for deletion after expiration :param ip_key: :param ip_value_info:

proxy/base_proxy.py:58

↓ 3 callersFunctionsum

(e, t)

libs/douyin.js:86

↓ 3 callersFunctionupdate_ks_video_comment

(video_id: str, comment_item: Dict)

store/kuaishou/__init__.py:90

↓ 2 callersMethod__init__

(self, **kwargs)

store/zhihu/_store_impl.py:61

↓ 2 callersMethod__init__

(self, **kwargs)

store/weibo/_store_impl.py:69

↓ 2 callersMethod__init__

(self, **kwargs)

store/tieba/_store_impl.py:62

↓ 2 callersMethod__init__

(self, **kwargs)

store/kuaishou/_store_impl.py:61

↓ 2 callersMethod__process_req_params

( self, uri: str, params: Optional[Dict] = None, headers: Optional[Dict] = Non

media_platform/douyin/client.py:71

↓ 2 callersMethod_api_user_link

(user: Dict)

media_platform/tieba/help.py:103

↓ 2 callersMethod_api_user_map

(api_data: Dict)

media_platform/tieba/help.py:99

↓ 2 callersFunction_assert_raw_values_absent

禁用的原始敏感值(uid/sec_uid/头像/签名/IP 等)不得出现在任何存储值里。 creator_hash 是由 uid 派生的匿名哈希(已单独断言 ≠ 原文),不参与子串扫描。

tests/test_douyin_no_user_info.py:152

↓ 2 callersFunction_build_aweme_item

贴近真实抖音结构的 mock aweme_item。含会被拍平的用户字段(uid/sec_uid/short_id/unique_id/avatar/signature)与 ip_label,这些必须不落库;同时含 statistics/video/music/images 等内容

tests/test_douyin_no_user_info.py:53

↓ 2 callersFunction_build_comment_item

贴近真实抖音结构的 mock comment_item。aweme_id 须与传入 update_dy_aweme_comment 的 aweme_id 一致。

tests/test_douyin_no_user_info.py:97

↓ 2 callersMethod_clean_title

(cls, title: str, tieba_name: str = "")

media_platform/tieba/help.py:321

↓ 2 callersMethod_create_browser_context

Create or get browser context

tools/cdp_browser.py:360

↓ 2 callersMethod_extract_answer_content

extract zhihu answer content Args: answer: zhihu answer Returns:

media_platform/zhihu/help.py:101

↓ 2 callersMethod_extract_api_content_text

(cls, content: Any)

media_platform/tieba/help.py:84

↓ 2 callersMethod_extract_article_content

extract zhihu article content Args: article: zhihu article Returns:

media_platform/zhihu/help.py:128

↓ 2 callersMethod_extract_content_list

extract zhihu content list Args: content_list: Returns:

media_platform/zhihu/help.py:77

↓ 2 callersMethod_extract_creator_portrait

(creator_url: str)

media_platform/tieba/client.py:172

↓ 2 callersMethod_extract_note_id_from_url

(url: str)

media_platform/tieba/help.py:63

↓ 2 callersMethod_extract_zvideo_content

extract zhihu zvideo content Args: zvideo: Returns:

media_platform/zhihu/help.py:155

↓ 2 callersFunction_filter_model_fields

只保留目标 ORM 模型已有的列，避免把已删除/多余字段（如 avatar/gender/ profile_url/ip_location/user_id）传给 ORM 构造而报错。教学版兜底保护。

store/weibo/_store_impl.py:61

↓ 2 callersMethod_get_browser_websocket_url

Get browser WebSocket connection URL

tools/cdp_browser.py:288

↓ 2 callersMethod_get_pc_page_data

(self, note_id: str, page: int = 1)

media_platform/tieba/client.py:150

↓ 2 callersMethod_get_pc_tbs

(self)

media_platform/tieba/client.py:133

↓ 2 callersFunction_on_signal

(signum: int)

tools/app_runner.py:69

↓ 2 callersMethod_parse_log_level

Parse log level

api/services/crawler_manager.py:80

↓ 2 callersMethod_test_cdp_connection

Test if CDP connection is available

tools/cdp_browser.py:227

↓ 2 callersFunctionarray_0_16_offset

(e)

libs/zhihu.js:43

↓ 2 callersMethodbatch_get_note_comments

Batch get note comments Args: note_detail_list: Returns:

media_platform/tieba/core.py:313

↓ 2 callersMethodbatch_get_notes_full_text

Batch get full text content of posts :param note_list: List of posts :return: Updated list of posts

media_platform/weibo/core.py:461

↓ 2 callersFunctioncheckEnvironment

()

webui/src/components/env/EnvironmentCheck.tsx:29

↓ 2 callersMethodcheck_is_img_path

(img, img_type, resize)

tools/slider_util.py:55

↓ 2 callersMethodcheck_login_state

Verify login status using dual-check: UI elements and Cookies.

media_platform/xhs/login.py:52

↓ 2 callersMethodcheck_page_display_slider

Check if slider verification appears on the page :return:

media_platform/douyin/login.py:171

↓ 2 callersFunctioncompare_schemas

Compare database structure with ORM model structure and return differences

test/test_db_sync.py:66

↓ 2 callersMethodcreate_store

()

store/weibo/__init__.py:48

↓ 2 callersMethodcreate_store

()

store/tieba/__init__.py:43

↓ 2 callersMethodcreate_store

()

store/kuaishou/__init__.py:47

↓ 2 callersMethodcreate_store

()

store/douyin/__init__.py:47

↓ 2 callersMethoddisconnect

(self, websocket: WebSocket)

api/routers/websocket.py:39

↓ 2 callersFunctione

(t)

media_platform/douyin/help.py:46

↓ 2 callersMethodextract_creator_info_from_api

Extract Tieba creator information from current PC creator JSON API.

media_platform/tieba/help.py:222

↓ 2 callersMethodextract_creator_thread_id_list_from_api

Extract creator thread ids from current PC creator feed JSON API.

media_platform/tieba/help.py:240

↓ 2 callersMethodextract_ip_and_pub_time

Extract IP location and publish time from HTML content Args: html_content: HTML string Returns: Tupl

media_platform/tieba/help.py:713

↓ 2 callersMethodextract_note_detail

Extract Tieba post details from post detail page Args: page_content: HTML string of page content Returns:

media_platform/tieba/help.py:498

↓ 2 callersMethodextract_offset

extract offset Args: paging_info: Returns:

media_platform/zhihu/help.py:277

↓ 2 callersMethodextract_search_note_list_from_api

Extract Tieba post list from current PC search JSON API.

media_platform/tieba/help.py:124

↓ 2 callersMethodextract_tieba_note_list

Extract Tieba post list from Tieba page Args: page_content: HTML string of page content Returns: Lis

media_platform/tieba/help.py:451

↓ 2 callersMethodextract_tieba_note_list_from_frs_api

Extract Tieba thread ids from current PC forum page JSON API. The by-forum command immediately fetches full details for every id, so

media_platform/tieba/help.py:252

↓ 2 callersMethodextract_tieba_note_parment_comments

Extract Tieba post first-level comments from comment page Args: page_content: HTML string of page content not

media_platform/tieba/help.py:559

↓ 2 callersMethodfind_many

Query multiple records (limit=0 means no limit)

database/mongodb_store_base.py:124

↓ 2 callersFunctionformatCellValue

(value: unknown)

webui/src/components/data/preview/DataPreviewTable.tsx:35

↓ 2 callersMethodget

Get the value of a key from the cache :param key: :return:

cache/local_cache.py:56

↓ 2 callersMethodget_all_notes_by_creator_url

Get all creator posts by current PC creator feed API.

media_platform/tieba/client.py:811

↓ 2 callersFunctionget_async_engine

(db_type: str = None)

database/db_session.py:53

↓ 2 callersMethodget_aweme_detail

Get note detail

media_platform/douyin/core.py:218

↓ 2 callersFunctionget_db_schema

Get current table structure of the database

test/test_db_sync.py:45

↓ 2 callersFunctionget_long_int

(round, long_str)

libs/douyin.js:227

↓ 2 callersMethodget_note_all_comments

Get all first-level comments for specified post (uses Playwright to access page, avoiding API detection) Args: note_detai

media_platform/tieba/client.py:452

↓ 2 callersMethodget_note_info_by_id

Get note details by note ID :param note_id: :return:

media_platform/weibo/client.py:258

↓ 2 callersMethodget_note_media

(self, url: str)

media_platform/xhs/client.py:207

↓ 2 callersMethodget_notes_by_keyword

Search Tieba posts by keyword (uses Playwright to access page, avoiding API detection) Args: keyword: Keyword

media_platform/tieba/client.py:375

↓ 2 callersMethodget_proxy

Randomly extract a proxy IP from the proxy pool :return:

proxy/proxy_ip_pool.py:100

↓ 2 callersFunctionget_search_id

()

media_platform/xhs/help.py:278

↓ 2 callersMethodget_specified_videos

get specified videos info from URLs or BV IDs :param video_url_list: List of video URLs or BV IDs :return:

media_platform/bilibili/core.py:387

↓ 2 callersMethodget_status

Get current status

api/services/crawler_manager.py:195

↓ 2 callersFunctionget_trace_id

(img_url: str)

media_platform/xhs/help.py:299

↓ 2 callersMethodget_video_info_task

Get video detail task

media_platform/kuaishou/core.py:209

↓ 2 callersMethodimage_edge_detection

(img)

tools/slider_util.py:126

↓ 2 callersMethodlaunch_browser

Launch browser and create browser context

media_platform/zhihu/core.py:424

Functions1,133 in github.com/NanmiCoder/MediaCrawler