Code
Hub
Workspaces
Following
Trending
Connect
MCP
copy
Create free account
hub
/
github.com/GAO-pooh/paper-scraper
/ functions
Functions
163 in github.com/GAO-pooh/paper-scraper
⨍
Functions
163
◇
Types & classes
6
↓ 17 callers
Function
send
(method, params=None)
sd_scraper.py:190
↓ 11 callers
Method
_is_chrome_debug_ready
检查 Chrome 调试端口是否已就绪。
sd_scraper.py:1320
↓ 11 callers
Function
send
(method, params=None)
sd_scraper_en.py:186
↓ 7 callers
Function
_input_optional
(prompt)
sd_scraper_en.py:1406
↓ 7 callers
Function
_input_optional
(prompt)
sd_scraper.py:1782
↓ 7 callers
Method
_is_chrome_debug_ready
Check whether the Chrome debug port is available.
sd_scraper_en.py:987
↓ 7 callers
Method
send
(method, params=None)
informs_scraper.py:626
↓ 6 callers
Method
_search
Two-step search: Step 1: Fetch search page HTML → get csrf_token cookie + searchToken Step 2: Call /search/api with token, pa
sd_scraper_en.py:404
↓ 6 callers
Method
_search
两步搜索流程: Step 1: 访问搜索页 HTML → 获取 csrf_token cookie + searchToken Step 2: 用 token 调 /search/api 获取 JSON 数据,支持分页
sd_scraper.py:419
↓ 5 callers
Method
_check_sd_access
检查当前 Chrome 是否有机构全文访问权限。 返回 True 表示有权限,False 表示需要登录(含 CARSI 重定向情形)。
sd_scraper.py:1483
↓ 5 callers
Method
_get
带重试、限速、浏览器导航头的 GET 请求。 若 curl_cffi 被 403 拦截,自动回退到 CDP(真实 Chrome)发请求。
informs_scraper.py:690
↓ 5 callers
Method
_get
GET request with retry, rate limiting, and browser navigation headers. Falls back to CDP (real Chrome) automatically on 403.
informs_scraper_en.py:658
↓ 5 callers
Method
_launch_chrome_with_debug
自动以调试模式启动 Chrome: 1. 把默认 Profile 的关键浏览器状态复制到临时目录 2. 用 --user-data-dir + --remote-debugging-port 启动 Chrome 3. 最多等待 40s
sd_scraper.py:1330
↓ 5 callers
Method
_warmup
访问主页让服务器建立干净 session,收集服务器下发的 XSRF cookie。
informs_scraper.py:549
↓ 5 callers
Method
_warmup
Visit the homepage to establish a clean session and collect XSRF cookies.
informs_scraper_en.py:528
↓ 4 callers
Method
_is_chrome_debug_ready
检查 Chrome 远程调试端口是否就绪。
informs_scraper.py:166
↓ 4 callers
Method
_is_chrome_debug_ready
(self)
informs_scraper_en.py:162
↓ 3 callers
Method
_delay
(self)
informs_scraper.py:535
↓ 3 callers
Method
_delay
(self)
informs_scraper_en.py:515
↓ 3 callers
Method
_launch_chrome_with_debug
Launch Chrome in debug mode: 1. Copy key browser state from the default Profile to a temp directory 2. Start Chrome with --us
sd_scraper_en.py:997
↓ 3 callers
Method
_load_browser_cookies
从本机 Chrome 读取 pubsonline.informs.org 的持久化 Cookie。 注意:browser_cookie3 只能读磁盘 cookie,不含 session cookie, 因此效果不如 CDP 提取(--chrome-l
informs_scraper.py:381
↓ 3 callers
Method
_load_browser_cookies
Read persistent pubsonline.informs.org cookies from local Chrome. Note: browser_cookie3 only reads disk cookies, not session cookies.
informs_scraper_en.py:372
↓ 3 callers
Method
_make_pdf_filename
生成 PDF 文件名:{序号}_{第一作者姓}_{年份}_{标题截断}.pdf
sd_scraper.py:706
↓ 3 callers
Method
save_to_csv
(self, results, filename, output_dir)
sd_scraper_en.py:648
↓ 3 callers
Method
save_to_csv
(self, results, filename, output_dir)
sd_scraper.py:669
↓ 2 callers
Method
_apply_cookie_header
(self)
sd_scraper_en.py:323
↓ 2 callers
Method
_apply_cookie_header
把 _cookie_dict 拼成 Cookie 请求头发送(比 session.cookies 更可靠)。
sd_scraper.py:328
↓ 2 callers
Method
_check_sd_access
Check whether the current Chrome session has institutional full-text access. Returns True if access is confirmed.
sd_scraper_en.py:1132
↓ 2 callers
Method
_delay
(self)
sd_scraper_en.py:367
↓ 2 callers
Method
_delay
(self)
sd_scraper.py:373
↓ 2 callers
Function
_dt_get_header
(headers, name: str)
sd_scraper_en.py:62
↓ 2 callers
Function
_dt_get_header
(headers, name: str)
sd_scraper.py:62
↓ 2 callers
Function
_dt_is_pdf_url
(url: str)
sd_scraper_en.py:75
↓ 2 callers
Function
_dt_is_pdf_url
(url: str)
sd_scraper.py:75
↓ 2 callers
Method
_fetch_search_token
Fetch the session cookie and searchToken required for the search API. Uses a clean session (no institutional cookies) to avoid sessio
sd_scraper_en.py:370
↓ 2 callers
Method
_fetch_search_token
用干净 session 访问搜索页面,获取: 1. 服务器下发的 session cookie(EUID、csrf_token 等) 2. searchToken(嵌入在页面 INITIAL_STATE 中) 注意:不带浏览器 co
sd_scraper.py:376
↓ 2 callers
Method
_get_cookies_via_browser_cookie3
从磁盘 Chrome Cookies 文件读取持久化 Cookie(备用,缺少 session cookies)。
sd_scraper.py:732
↓ 2 callers
Method
_get_cookies_via_cdp
从 CDP Chrome 上下文提取所有相关 Cookie(含 session cookies)。
sd_scraper.py:717
↓ 2 callers
Method
_get_total_results
从搜索结果页提取总结果数。
informs_scraper.py:978
↓ 2 callers
Method
_get_total_results
Extract total result count from a search page.
informs_scraper_en.py:921
↓ 2 callers
Method
_make_pdf_filename
Generate filename: {index}_{first_author_surname}_{year}_{title}.pdf
sd_scraper_en.py:685
↓ 2 callers
Method
_parse_article_card
从 BeautifulSoup Tag 解析单篇文章元数据。 优先使用 INFORMS PubsOnLine 的真实 class 名,备用通用选择器。
informs_scraper.py:753
↓ 2 callers
Method
_parse_article_card
Parse a single article card from BeautifulSoup.
informs_scraper_en.py:715
↓ 2 callers
Method
_parse_search_html
解析搜索结果页 HTML,返回文章列表。
informs_scraper.py:877
↓ 2 callers
Method
_parse_search_html
Parse search result page HTML and return a list of article dicts.
informs_scraper_en.py:825
↓ 2 callers
Method
_parse_toc_html
解析期刊 TOC(目录)页 HTML。结构与搜索结果类似但略有不同。
informs_scraper.py:948
↓ 2 callers
Method
_parse_toc_html
Parse a journal TOC page HTML.
informs_scraper_en.py:892
↓ 2 callers
Method
_wait_for_login
打开 ScienceDirect 文章页(触发 CARSI 自动重定向),提示用户完成机构登录, 等待用户在终端按 Enter 确认。
sd_scraper.py:1514
↓ 2 callers
Method
download_pdfs_devtools
Download PDFs via Chrome DevTools Protocol (pure websocket-client). How it works: 1. Connect to a running Chrome debug insta
sd_scraper_en.py:729
↓ 2 callers
Method
download_pdfs_devtools
通过 Chrome DevTools Protocol 下载 PDF(纯 websocket-client,无需 Playwright)。 工作原理: 1. 连接已运行的 Chrome 调试实例(端口 9222) 2. 逐篇在新标签
sd_scraper.py:1043
↓ 2 callers
Method
save_to_csv
(self, results: list, filename: str, output_dir: str)
informs_scraper.py:1267
↓ 2 callers
Method
save_to_csv
(self, results: list, filename: str, output_dir: str)
informs_scraper_en.py:1204
↓ 2 callers
Method
save_to_json
(self, results, filename, output_dir)
sd_scraper_en.py:658
↓ 2 callers
Method
save_to_json
(self, results, filename, output_dir)
sd_scraper.py:679
↓ 2 callers
Method
save_to_xlsx
(self, results, filename, output_dir)
sd_scraper_en.py:666
↓ 2 callers
Method
save_to_xlsx
(self, results, filename, output_dir)
sd_scraper.py:687
↓ 2 callers
Method
search_advanced
Advanced search: combine multiple criteria.
sd_scraper_en.py:620
↓ 2 callers
Method
search_advanced
高级搜索:组合多个条件。
sd_scraper.py:641
↓ 2 callers
Method
search_by_author
Search by author name.
sd_scraper_en.py:606
↓ 2 callers
Method
search_by_author
按作者姓名搜索。
sd_scraper.py:627
↓ 2 callers
Method
search_by_issn
Search by journal ISSN.
sd_scraper_en.py:612
↓ 2 callers
Method
search_by_issn
按期刊 ISSN 搜索。
sd_scraper.py:633
↓ 2 callers
Method
search_by_journal
Browse a journal by name.
sd_scraper_en.py:588
↓ 2 callers
Method
search_by_journal
按期刊名称浏览。
sd_scraper.py:609
↓ 2 callers
Method
search_by_journal_keyword
Search by keyword within a specific journal.
sd_scraper_en.py:597
↓ 2 callers
Method
search_by_journal_keyword
在指定期刊内按关键词搜索。
sd_scraper.py:618
↓ 2 callers
Method
search_by_keyword
Search by keyword (supports Boolean operators AND / OR / NOT).
sd_scraper_en.py:577
↓ 2 callers
Method
search_by_keyword
按关键词搜索(支持布尔运算符 AND / OR / NOT)。
sd_scraper.py:598
↓ 1 callers
Function
_build_parser
()
informs_scraper.py:1431
↓ 1 callers
Function
_build_parser
()
informs_scraper_en.py:1367
↓ 1 callers
Method
_cdp_fetch_html
通过 CDP 在调试 Chrome 里打开 url,等待 JS 渲染后返回完整 HTML。 策略:导航 → 等 loadEventFired → 再等 wait_seconds(JS 渲染)→ 取 HTML。 全程只维护一个消息循环,无嵌套轮询,逻
informs_scraper.py:576
↓ 1 callers
Method
_cdp_fetch_html
Open url in debug Chrome via CDP, wait for JS rendering, and return full HTML.
informs_scraper_en.py:552
↓ 1 callers
Method
_chrome_login_flow
完整的 Chrome 登录流程: 1. 启动调试 Chrome(复用你的默认 Profile) 2. 打开 INFORMS 登录页 3. 等你手动登录,按 Enter 确认 4. 通过 CDP 提取 session c
informs_scraper.py:326
↓ 1 callers
Method
_chrome_login_flow
Full Chrome login flow: 1. Launch debug Chrome (reuses your default Profile) 2. Open INFORMS login page 3. Wait for y
informs_scraper_en.py:319
↓ 1 callers
Function
_dt_capture_pdf
Navigate to url in an existing DevTools tab and capture PDF bytes via Network/Fetch interception. Returns (bytes | None, note_str). n
sd_scraper_en.py:168
↓ 1 callers
Function
_dt_capture_pdf
在已有 DevTools 标签页中导航到 url,通过 Network/Fetch 拦截捕获 PDF 字节。 返回 (bytes | None, note_str)。 note_str 以 "blocked:" 开头表示触发了反爬封锁页面。
sd_scraper.py:173
↓ 1 callers
Function
_dt_is_pdf_fetch_response
(url: str, status, headers)
sd_scraper_en.py:86
↓ 1 callers
Function
_dt_is_pdf_fetch_response
(url: str, status, headers)
sd_scraper.py:86
↓ 1 callers
Function
_dt_is_pdf_response
(response: dict)
sd_scraper_en.py:80
↓ 1 callers
Function
_dt_is_pdf_response
(response: dict)
sd_scraper.py:80
↓ 1 callers
Method
_extract_cookies_via_cdp
通过 CDP websocket 从运行中的 Chrome 提取 pubsonline.informs.org 的所有 cookie。 返回拼好的 Cookie 请求头字符串。无需 Playwright。
informs_scraper.py:240
↓ 1 callers
Method
_extract_cookies_via_cdp
Extract pubsonline.informs.org cookies from running Chrome via CDP websocket. Returns a formatted Cookie header string. No Playwright
informs_scraper_en.py:234
↓ 1 callers
Method
_get_via_cdp
curl_cffi 被 403 时的备用方案:让调试 Chrome 发请求,返回一个模拟 Response。 搜索结果页是 JS 渲染,传入 wait_selector 等待结果容器出现。 同时保存原始 HTML 到临时文件方便排查选择器问题。
informs_scraper.py:722
↓ 1 callers
Method
_get_via_cdp
Fallback when curl_cffi is blocked (403): let debug Chrome make the request. Returns a mock response object with .status_code and .te
informs_scraper_en.py:689
↓ 1 callers
Function
_input_int
(prompt, default)
sd_scraper_en.py:1399
↓ 1 callers
Function
_input_int
(prompt, default)
sd_scraper.py:1775
↓ 1 callers
Function
_interactive_mode
交互式向导(无参数运行时启动)。
informs_scraper.py:1517
↓ 1 callers
Function
_interactive_mode
Interactive wizard (launched when no arguments are given).
informs_scraper_en.py:1452
↓ 1 callers
Method
_launch_chrome_with_debug
以调试模式启动 Chrome,复用默认 Profile(带已保存的密码/书签/历史)。 返回 Popen 对象,失败返回 None。
informs_scraper.py:176
↓ 1 callers
Method
_launch_chrome_with_debug
Launch Chrome in debug mode, reusing the default Profile (saved passwords, bookmarks, history). Returns Popen object, or None
informs_scraper_en.py:171
↓ 1 callers
Method
_load_browser_cookies
Read sciencedirect.com cookies directly from the local Chrome installation.
sd_scraper_en.py:329
↓ 1 callers
Method
_load_browser_cookies
直接从本机 Chrome 读取 sciencedirect.com 的 cookie。
sd_scraper.py:335
↓ 1 callers
Method
_load_cookies
Load cookies from a JSON file.
sd_scraper_en.py:347
↓ 1 callers
Method
_load_cookies
从 JSON 文件加载 cookies。
sd_scraper.py:353
↓ 1 callers
Method
_load_cookies_file
从 JSON 文件(Netscape 或字典格式)加载 Cookie。
informs_scraper.py:415
↓ 1 callers
Method
_load_cookies_file
Load cookies from a JSON file (Netscape or dict format).
informs_scraper_en.py:406
↓ 1 callers
Method
_login
用会员账号密码登录 INFORMS PubsOnLine(Atypon 标准流程)。 步骤: 1. GET 登录页,提取 CSRF token 2. POST 凭证到 /action/doLogin 3.
informs_scraper.py:433
↓ 1 callers
Method
_login
Log in with member credentials via the standard Atypon flow: 1. GET login page, extract CSRF token 2. POST credentials to /ac
informs_scraper_en.py:424
↓ 1 callers
Method
_make_pdf_filename
生成文件名:{序号}_{第一作者姓}_{年份}_{标题截断}.pdf
informs_scraper.py:1304
↓ 1 callers
Method
_make_pdf_filename
Generate filename: {index}_{first_author_surname}_{year}_{title}.pdf
informs_scraper_en.py:1241
next →
1–100 of 163, ranked by callers