Function ParseRobotsTxt

robots/robots.go:34–60 · view source on GitHub ↗

(text []byte)

Source from the content-addressed store, hash-verified

source not stored for this graph (policy: none)

GetRobotsForURLFunction · 0.85

loadTestRobotsFunction · 0.85

newRobotsGroupFunction · 0.85

trimSpaceAndCommentsFunction · 0.85

splitLineFunction · 0.85

loadTestRobotsFunction · 0.68