MCPcopy Create free account
hub / github.com/apify/crawlee-python / robots_txt

Function robots_txt

tests/unit/server.py:415–417  ·  view source on GitHub ↗

Handle requests for the robots.txt file.

(_scope: dict[str, Any], _receive: Receive, send: Send)

Source from the content-addressed store, hash-verified

413
414
415async def robots_txt(_scope: dict[str, Any], _receive: Receive, send: Send) -> None:
416 """Handle requests for the robots.txt file."""
417 await send_html_response(send, ROBOTS_TXT)
418
419
420async def get_compressed(_scope: dict[str, Any], _receive: Receive, send: Send) -> None:

Callers

nothing calls this directly

Calls 1

send_html_responseFunction · 0.85

Tested by

no test coverage detected