Type f is a simple syntax helper to create test cases' extension functions.

S is a convenience type definition, it is a map[string]interface{} that can be used to enqueue URLs (the string) with state information.

U is a convenience type definition, it is a map[*url.URL]interface{} that can be used to enqueue URLs with state information.

CrawlError contains information about the crawling error.

CrawlErrorKind indicated the kind of crawling error.

Crawler is the web crawler that processes URLs and manages the workers.

DefaultExtender is a default working implementation of an extender. It is possible to nest such a value in a custom struct so that only the Extender m

DelayInfo contains the delay configuration: the Options delay, the Robots.txt delay, and the last delay used.

Create the Extender implementation, based on the gocrawl-provided DefaultExtender, because we don't want/need to override all methods.

Extender defines the extension methods required by the crawler.

FetchInfo contains the fetch information: the duration of the fetch, the returned status code, whether or not it was a HEAD request, and whether or no

LogFlags is a set of flags that control the logging of the Crawler.

Options contains the configuration for a Crawler to customize the crawling process.

URLContext contains all information related to an URL to process.

Type a is a simple syntax helper to create test cases' asserts.

Extension method enum

The file fetcher, that loads URLs from files in the testdata/ directory.

Type and unique value to identify a called-with argument position to ignore.

The pop channel is a stacked channel used by workers to pop the next URL(s) to process.

The spy extender adds counting the number of calls for each extender method, and tracks the log so that it can be asserted.

Test case structure.

The worker is dedicated to fetching and visiting a given host, respecting this host's robots.txt crawling policies.

Communication from worker to the master crawler, about the crawling of a URL