MCPcopy
hub / github.com/apify/crawlee / load

Method load

packages/utils/src/internals/sitemap.ts:416–426  ·  view source on GitHub ↗

* Fetch sitemap content from given URL or URLs and return URLs of referenced pages. * @param urls sitemap URL(s) * @param proxyUrl URL of a proxy to be used for fetching sitemap contents

(
        urls: string | string[],
        proxyUrl?: string,
        parseSitemapOptions?: ParseSitemapOptions,
    )

Source from the content-addressed store, hash-verified

414 * @param proxyUrl URL of a proxy to be used for fetching sitemap contents
415 */
416 static async load(
417 urls: string | string[],
418 proxyUrl?: string,
419 parseSitemapOptions?: ParseSitemapOptions,
420 ): Promise<Sitemap> {
421 return await this.parse(
422 (Array.isArray(urls) ? urls : [urls]).map((url) => ({ type: 'url', url })),
423 proxyUrl,
424 parseSitemapOptions,
425 );
426 }
427
428 /**
429 * Parse XML sitemap content from a string and return URLs of referenced pages. If the sitemap references other sitemaps, they will be loaded via HTTP.

Callers 15

sitemap.test.tsFile · 0.45
htmlToTextFunction · 0.45
tryCommonNamesMethod · 0.45
parseHandlesFromHtmlFunction · 0.45
parseWithCheerioFunction · 0.45
_runRequestHandlerMethod · 0.45
parseWithCheerioFunction · 0.45
constructorMethod · 0.45
_runRequestHandlerMethod · 0.45
_runRequestHandlerMethod · 0.45
cheerio.test.tsFile · 0.45

Calls 2

parseMethod · 0.95
mapMethod · 0.80

Tested by

no test coverage detected