Helper to update output fields with mutex lock

testR returns a shared router so we pay the Redis-timeout cost only once.

hostOnly returns scheme://host from a URL, used to keep full URLs out of logs.

cachedTask executes a task with caching support Returns true if value was found in cache, false otherwise

---------- Helpers ---------- isAllowedRune checks if rune is a-z, 0-9, or hyphen (we lowercase runes first at call site)

nodeText returns concatenated text of a node subtree

parseWhoisDate attempts to parse WHOIS date strings into time.Time

safeBase returns a minimal Response that should produce verdict "Safe". All boolean bad-signals are off; rank is top-10k; ICANN TLD; valid NS+MX.

ssrfSafeDialContext wraps a Dialer's DialContext to reject connections to private/reserved IP ranges, preventing SSRF attacks. DNS is resolved once he

TakeScreenshot takes a screenshot and returns the image bytes It first checks if a cached screenshot exists, otherwise takes a new one

extractSLD returns just the second-level domain (SLD) of a hostname, stripping both subdomains and the public suffix. e.g. "www.paypal.com" → "paypal"

fetchRDAPWithContext queries RDAP with context support and timeout.

Helper functions for environment variables

newSafeHTTPClient returns an *http.Client that blocks SSRF. Use for plain fetches that don't need a custom CheckRedirect or transport.

newSafeTransport returns an *http.Transport with SSRF-safe DialContext. Use when building a custom http.Client (e.g. one with a CheckRedirect hook).

parseBoolField understands JSON bool and legacy string "y"/"n".

sanitizeURL creates a filesystem-safe filename from a URL

shannonEntropy computes Shannon entropy in bits over runes

ConvertTimings converts map of timings to a sorted slice (desc by duration)

Expire sets an expiration on a key

FlushAll removes all keys from the current database

GetScreenshotPath returns the file path for a screenshot based on normalized URL

GetService returns the default service instance (singleton)

Increment increments a key and returns the new value

LookupWithContext tries RDAP first with timeout, falls back to WHOIS if RDAP fails.

NewService creates a new screenshot service with a shared browser allocator

Scan returns all keys matching the given glob pattern using SCAN (non-blocking).

TTL returns the remaining time-to-live for a key. Returns -1 if no expiry, -2 if key missing.

TakeScreenshot is a convenience function that uses the default service

allowedOrigins reads CORS_ALLOWED_ORIGINS (comma-separated) from the environment.

bigramEnglishiness: fraction of letter-only bigrams that occur in a small English bigram set

buildPhishingResult converts the internal PhishTankResult into the public PhishingResult. Returns nil when no check was performed (cache miss + fetch

extractFormInfo inspects a <form> node and returns a FormInfo

extractSLD returns the second-level label if available (example.com -> "example") keeps behavior simple (does not handle public suffix list). That's i

getEnv gets an environment variable or returns a default value

getRDAPServer returns the RDAP base URL for a TLD. It checks the well-known map first, then falls back to the IANA bootstrap.

levenshtein computes the edit distance between two strings.

loadIANABootstrap fetches and parses the IANA RDAP bootstrap file. Results are cached in memory for 24 hours; stale data is returned on error.

loadTimezone reads LOG_TIMEZONE and returns the matching *time.Location. Falls back to UTC on unknown values.

normalizeURL normalizes a URL for consistent cache keys - Lowercase host - Remove fragment - Sort query params - Trim trailing slash (except root)

runTasks runs all tasks in parallel, collects timings and non-fatal errors

helpers

validateURL ensures the URL is safe to navigate to

verifyArgon2id checks a password against a PHC-format Argon2id hash: $argon2id$v=19$m=65536,t=3,p=4$<base64_salt>$<base64_hash>

AdminErrorsHandler returns the last N error/panic records (newest first).

AdminLoginHandler validates the admin password against the stored Argon2id hash and returns a signed session token on success.

AdminRecentHandler returns the last N scan records (newest first).

AdminStatsHandler returns live scan metrics derived from the in-memory ring buffer.

Analyze runs all tasks and builds the final response