MCPcopy Create free account
hub / github.com/apify/crawlee-python / _get_origin_url

Function _get_origin_url

src/crawlee/_utils/sitemap.py:219–226  ·  view source on GitHub ↗

Determine the origin URL for a sitemap source.

(source: SitemapSource)

Source from the content-addressed store, hash-verified

217
218
219def _get_origin_url(source: SitemapSource) -> str:
220 """Determine the origin URL for a sitemap source."""
221 if source['type'] == 'url' and 'url' in source:
222 return source['url']
223 if source['type'] == 'raw' and 'content' in source:
224 # For raw content sources, create a consistent identifier
225 return f'raw://{sha256(source["content"].encode()).hexdigest()}'
226 return ''
227
228
229async def _process_sitemap_item(

Callers 1

_process_sitemap_itemFunction · 0.85

Calls

no outgoing calls

Tested by

no test coverage detected