MCPcopy Create free account
hub / github.com/Matir/webborer / AddRobotsFilter

Method AddRobotsFilter

filter/filter.go:90–105  ·  view source on GitHub ↗

Filter data from robots.txt

(scope []*url.URL, clientFactory client.ClientFactory)

Source from the content-addressed store, hash-verified

source not stored for this graph (policy: none)

Callers 3

mainFunction · 0.95
TestRobotsFilter_SuccessFunction · 0.95
TestRobotsFilter_FailFunction · 0.95

Calls 4

FilterURLMethod · 0.95
LogfFunction · 0.92
GetRobotsForURLFunction · 0.92
GetForUserAgentMethod · 0.80

Tested by 2

TestRobotsFilter_SuccessFunction · 0.76
TestRobotsFilter_FailFunction · 0.76