hub / github.com/FlowiseAI/Flowise / crawl

Function crawl

packages/components/src/utils.ts:439–478 · view source on GitHub ↗

* Recursive crawl using normalizeURL and getURLsFromHTML * @param {string} baseURL * @param {string} currentURL * @param {string[]} pages * @param {number} limit * @returns {Promise }

(baseURL: string, currentURL: string, pages: string[], limit: number)

Source from the content-addressed store, hash-verified

source not stored for this graph (policy: none)

Callers 1

webCrawlFunction · 0.85

Calls 4

secureFetchFunction · 0.90

normalizeURLFunction · 0.85

getURLsFromHTMLFunction · 0.85

getMethod · 0.45

Tested by

no test coverage detected