Functions — github.com/Sophomoresty/bpc-fetch

hub / github.com/Sophomoresty/bpc-fetch / functions

Functions83 in github.com/Sophomoresty/bpc-fetch

⨍ Functions 83 ◇ Types & classes 2

↓ 10 callersFunction_text

(el, tag: str, ns: dict | None = None)

src/bpc_fetch/discover.py:214

↓ 7 callersFunction_skip_ws

(text: str, i: int, length: int)

src/bpc_fetch/sites.py:140

↓ 6 callersFunctionget_sites_map

Get or build the sites strategy map. Caches to JSON for speed.

src/bpc_fetch/sites.py:259

↓ 5 callersFunction_has_content

Check if HTML has meaningful content (not just a paywall/redirect).

src/bpc_fetch/strategy.py:176

↓ 5 callersMethodbypass_type

(self)

src/bpc_fetch/sites.py:35

↓ 5 callersFunctiondomain_from_url

Extract registrable domain from URL.

src/bpc_fetch/sites.py:274

↓ 4 callersMethodstart

(self)

src/bpc_fetch/browser.py:41

↓ 4 callersMethodstop

(self)

src/bpc_fetch/browser.py:45

↓ 3 callersFunction_image_filename

(url: str, index: int)

src/bpc_fetch/extract.py:204

↓ 3 callersFunction_parse_date

(date_str: str | None)

src/bpc_fetch/discover.py:219

↓ 3 callersFunction_read_string

(text: str, i: int, length: int)

src/bpc_fetch/sites.py:150

↓ 3 callersFunction_read_value

Read a JS value: string, number, bool, regex, array.

src/bpc_fetch/sites.py:164

↓ 3 callersFunctionarticle_to_markdown

Convert extracted article to markdown with frontmatter.

src/bpc_fetch/extract.py:66

↓ 3 callersFunctiondownload_images

Download images to out_dir. Returns list of saved paths.

src/bpc_fetch/extract.py:109

↓ 3 callersFunctionextract_article

Extract article content from HTML. Returns dict with title, text, markdown, images. If dom_result is provided (from browser DOM extraction), use

src/bpc_fetch/extract.py:14

↓ 3 callersFunctionfetch_with_retries

Try primary strategy, fallback to googlebot, then browser, then archive.org. use_browser: True=force browser, False=skip browser, None=auto (bloc

src/bpc_fetch/strategy.py:79

↓ 3 callersFunctionget_db

(db_path: Path | None = None)

src/bpc_fetch/history.py:8

↓ 3 callersMethodpage

(self)

src/bpc_fetch/browser.py:52

↓ 2 callersFunction_build_route_patterns

Convert BPC block_regex to Playwright route glob patterns.

src/bpc_fetch/browser.py:72

↓ 2 callersFunction_build_strategy

(domain: str, name: str, props: dict)

src/bpc_fetch/sites.py:81

↓ 2 callersFunction_escape_yaml

(s: str)

src/bpc_fetch/extract.py:213

↓ 2 callersFunction_format

(item: dict, domain: str)

src/bpc_fetch/search.py:70

↓ 2 callersFunction_is_article_url

Heuristic: URL looks like an article (has date pattern or article path).

src/bpc_fetch/discover.py:202

↓ 2 callersFunction_is_paywalled

Detect if page shows paywall truncation markers.

src/bpc_fetch/strategy.py:196

↓ 2 callersFunction_parse_sitemap

Parse sitemap XML.

src/bpc_fetch/discover.py:135

↓ 2 callersFunction_read_object

(text: str, i: int, length: int)

src/bpc_fetch/sites.py:229

↓ 2 callersFunction_slugify

(text: str)

src/bpc_fetch/cli.py:406

↓ 2 callersFunctiondiscover

Discover recent articles from domain. Returns {ok, domain, count, articles}.

src/bpc_fetch/discover.py:32

↓ 2 callersFunctionfetch_page

Fetch page HTML with bypass headers. Returns (html, status_code).

src/bpc_fetch/strategy.py:65

↓ 2 callersFunctionparse_since

Parse --since value: 'today', 'Nd', or 'YYYY-MM-DD'.

src/bpc_fetch/discover.py:18

↓ 2 callersFunctionsearch_sites

Search supported sites. Uses Brave API if available, otherwise returns empty.

src/bpc_fetch/search.py:60

↓ 1 callersFunction_clean_paywall_text

Remove paywall/login prompts that leak into extracted text.

src/bpc_fetch/extract.py:217

↓ 1 callersFunction_cmd_batch

(args)

src/bpc_fetch/cli.py:348

↓ 1 callersFunction_cmd_crawl

(args)

src/bpc_fetch/cli.py:226

↓ 1 callersFunction_cmd_discover

(args)

src/bpc_fetch/cli.py:220

↓ 1 callersFunction_cmd_doctor

(args)

src/bpc_fetch/cli.py:152

↓ 1 callersFunction_cmd_fetch

(args)

src/bpc_fetch/cli.py:299

↓ 1 callersFunction_cmd_history

(args)

src/bpc_fetch/cli.py:244

↓ 1 callersFunction_cmd_install_browser

(args)

src/bpc_fetch/cli.py:212

↓ 1 callersFunction_cmd_search

(args)

src/bpc_fetch/cli.py:269

↓ 1 callersFunction_cmd_sites

(args)

src/bpc_fetch/cli.py:250

↓ 1 callersFunction_default_sites_js

Locate sites.js: PyInstaller bundle → package data → home fallback.

src/bpc_fetch/sites.py:9

↓ 1 callersFunction_dispatch

(args)

src/bpc_fetch/cli.py:130

↓ 1 callersFunction_emit_progress

(current: int, total: int, url: str)

src/bpc_fetch/crawl.py:126

↓ 1 callersFunction_enrich_result

Add next_command and recovery_command to all results.

src/bpc_fetch/cli.py:96

↓ 1 callersFunction_extract_author

(metadata: str | None)

src/bpc_fetch/extract.py:149

↓ 1 callersFunction_extract_date

(metadata: str | None)

src/bpc_fetch/extract.py:156

↓ 1 callersFunction_extract_entries

Extract site entries from JS object literal text. Handles regex literals, arrays, strings, numbers, booleans. Returns {site_name: {key: value

src/bpc_fetch/sites.py:96

↓ 1 callersFunction_extract_image_urls

Extract article images only — skip UI, nav, footer, sidebar images.

src/bpc_fetch/extract.py:163

↓ 1 callersFunction_extract_title

(html: str, metadata: str | None)

src/bpc_fetch/extract.py:139

↓ 1 callersFunction_fetch_one

(item: dict, idx: int)

src/bpc_fetch/crawl.py:70

↓ 1 callersFunction_fetch_one

(url: str)

src/bpc_fetch/cli.py:370

↓ 1 callersFunction_has_full_article

Check if HTML likely contains a full article (not just first paragraph).

src/bpc_fetch/strategy.py:188

↓ 1 callersFunction_parse_rss

Parse RSS/Atom XML into article list.

src/bpc_fetch/discover.py:84

↓ 1 callersFunction_parse_sitemap_index

Parse sitemap index, fetch most recent sub-sitemap.

src/bpc_fetch/discover.py:154

↓ 1 callersFunction_read_array

(text: str, i: int, length: int)

src/bpc_fetch/sites.py:213

↓ 1 callersFunction_read_regex

(text: str, i: int, length: int)

src/bpc_fetch/sites.py:191

↓ 1 callersFunction_regex_to_glob

Best-effort convert a simple regex fragment to a glob pattern.

src/bpc_fetch/browser.py:88

↓ 1 callersFunction_slugify

(text: str)

src/bpc_fetch/crawl.py:132

↓ 1 callersFunction_try_browser_homepage

Use Playwright to load homepage and extract article links (Cloudflare bypass).

src/bpc_fetch/discover.py:234

↓ 1 callersFunction_try_homepage

Extract article links from homepage.

src/bpc_fetch/discover.py:177

↓ 1 callersFunction_try_rss

Try common RSS feed paths.

src/bpc_fetch/discover.py:71

↓ 1 callersFunction_try_sitemap

Try sitemap.xml for recent URLs.

src/bpc_fetch/discover.py:120

↓ 1 callersFunctionbuild

()

packaging/build_win.py:7

↓ 1 callersFunctionbuild_fallback_headers

Fallback: Googlebot UA + Google referer.

src/bpc_fetch/strategy.py:55

↓ 1 callersFunctionbuild_headers

Build HTTP headers based on the site's bypass strategy.

src/bpc_fetch/strategy.py:18

↓ 1 callersFunctioncrawl

Search + discover + time filter + batch fetch. 1. If sites_filter given: discover from those sites, filter by since 2. Else: search query acr

src/bpc_fetch/crawl.py:17

↓ 1 callersFunctionexpandSiteRules

(sites, updated = false)

data/sites.js:3779

↓ 1 callersFunctionextract_article_dom

Extract article content directly from page DOM. More reliable than trafilatura for JS-rendered pages.

src/bpc_fetch/browser.py:174

↓ 1 callersFunctionfilter_urls

Filter a list of URLs to only those on supported paywall sites.

src/bpc_fetch/search.py:10

↓ 1 callersFunctionget_history

(domain: str | None = None, limit: int = 50, db_path: Path | None = None)

src/bpc_fetch/history.py:40

↓ 1 callersFunctioninit_custom_flex_domains

()

data/sites.js:3834

↓ 1 callersFunctionis_fetched

(url: str, db_path: Path | None = None)

src/bpc_fetch/history.py:23

↓ 1 callersFunctionmain

()

src/bpc_fetch/cli.py:11

↓ 1 callersFunctionmark_fetched

(url: str, domain: str, title: str, path: str, db_path: Path | None = None)

src/bpc_fetch/history.py:30

↓ 1 callersFunctionparse_sites_js

Parse sites.js and return {domain: SiteStrategy} map.

src/bpc_fetch/sites.py:54

↓ 1 callersFunctionsearch_brave

Search via Brave Search API. Requires BRAVE_API_KEY env var.

src/bpc_fetch/search.py:24

Method__aenter__

(self)

src/bpc_fetch/browser.py:64

Method__aexit__

(self, *_)

src/bpc_fetch/browser.py:68

Method__init__

(self, max_contexts: int = 3)

src/bpc_fetch/browser.py:35

Functionensure_browser

Check if Playwright Chromium is installed. Returns status dict.

src/bpc_fetch/browser.py:14

Functionfetch_with_browser

Fetch page using Playwright, blocking paywall scripts via route.

src/bpc_fetch/browser.py:105

Methodto_dict

(self)

src/bpc_fetch/sites.py:48