gocrawl

gocrawl is a polite, slim and concurrent web crawler written in Go.

For a simpler yet more flexible web crawler written in a more idiomatic Go style, you may want to take a look at fetchbot, a package that builds on the experience of gocrawl.

Translations

• 한국어(korean): https://github.com/PuerkitoBio/gocrawl/blob/master/doc/ko/README.md

Features

• Full control over the URLs to visit, inspect and query (using a pre-initialized [goquery][] document)
• Crawl delays applied per host
• Obedience to robots.txt rules (using the [robotstxt.go][robots] library)
• Concurrent execution using goroutines
• Configurable logging
• Open, customizable design providing hooks into the execution logic

Installation and dependencies

gocrawl depends on the following userland libraries:

• [goquery][]
• [purell][]
• [robotstxt.go][robots]

It requires Go1.1+ because of its indirect dependency on golang.org/x/net/html. To install:

go get github.com/PuerkitoBio/gocrawl

To install a previous version, you have to git clone https://github.com/PuerkitoBio/gocrawl into your $GOPATH/src/github.com/PuerkitoBio/gocrawl/ directory, and then run (for example) git checkout v0.3.2 to checkout a specific version, and go install to build and install the Go package.

Changelog

• 2019-07-22 : Use pre-compiled matchers for goquery (thanks @mikefaraponov). Tag v1.0.1.
• 2016-11-20 : Fix log message so that it prints enqueued URLs (thanks @oherych). Tag as v1.0.0.
• 2016-05-24 : Set the *URLContext.SourceURL() and *URLContext.NormalizedSourceURL() to the original URL on redirections (see [#55][i55]). Thanks to github user [@tmatsuo][tmatsuo].
• 2016-02-24 : Always use Options.UserAgent to make requests, use Options.RobotUserAgent only for robots.txt policy matching. Lint and vet the code a bit, better godoc documentation.
• 2014-11-06 : Change import paths of net/html to golang.org/x/net/html (see https://groups.google.com/forum/#!topic/golang-nuts/eD8dh3T9yyA).
• v0.4.1 : now go-getable, since goquery is go-getable too.
• v0.4.0 : BREAKING CHANGES major refactor, API changes:
  • Use an *URLContext structure as first argument to all Extender interface functions that are called in the context of an URL, instead of a simple *url.URL pointer that was sometimes normalized, sometimes not.
  • Remove the EnqueueOrigin enumeration flag. It wasn't even used by gocrawl, and it is a kind of state associated with the URL, so this feature is now generalized with the next bullet...
  • Add a State for each URL, so that the crawling process can associate arbitrary data with a given URL (for example, the ID of a record in the database). Fixes [issue #14][i14].
  • More idiomatic use of errors (ErrXxx variables, Run() now returns an error, removing the need for the EndReason enum).
  • Much simplified Filter() function. It now only returns a bool indicating if it should be visited or not. The HEAD request override feature is provided by the *URLContext structure, and can be set anywhere. The priority feature was unimplemented and has been removed from the return values, if it gets implemented it will probably be via the *URLContext structure too.
  • Start, Run, Visit, Visited and the EnqueueChan all work with the empty interface type for the URL data. While this is an inconvenience for compile-time checks, it allows for more flexibility regarding the state feature. Instead of always forcing a map[string]interface{} type even when no state is needed, gocrawl supports various types.
  • Some other internal changes, better tests.
• v0.3,2 : Fix the high CPU usage when waiting for a crawl delay.
• v0.3.1 : Export the HttpClient variable used by the default Fetch() implementation (see [issue #9][i9]).
• v0.3.0 : BEHAVIOR CHANGE filter done with normalized URL, fetch done with original, non-normalized URL (see [issue #10][i10]).
• v0.2.0 : BREAKING CHANGES rework extension/hooks.
• v0.1.0 : Initial release.

Example

From example_test.go:

// Only enqueue the root and paths beginning with an "a"
var rxOk = regexp.MustCompile(`http://duckduckgo\.com(/a.*)?$`)

// Create the Extender implementation, based on the gocrawl-provided DefaultExtender,
// because we don't want/need to override all methods.
type ExampleExtender struct {
    gocrawl.DefaultExtender // Will use the default implementation of all but Visit and Filter
}

// Override Visit for our need.
func (x *ExampleExtender) Visit(ctx *gocrawl.URLContext, res *http.Response, doc *goquery.Document) (interface{}, bool) {
    // Use the goquery document or res.Body to manipulate the data
    // ...

    // Return nil and true - let gocrawl find the links
    return nil, true
}

// Override Filter for our need.
func (x *ExampleExtender) Filter(ctx *gocrawl.URLContext, isVisited bool) bool {
    return !isVisited && rxOk.MatchString(ctx.NormalizedURL().String())
}

func ExampleCrawl() {
    // Set custom options
    opts := gocrawl.NewOptions(new(ExampleExtender))

    // should always set your robot name so that it looks for the most
    // specific rules possible in robots.txt.
    opts.RobotUserAgent = "Example"
    // and reflect that in the user-agent string used to make requests,
    // ideally with a link so site owners can contact you if there's an issue
    opts.UserAgent = "Mozilla/5.0 (compatible; Example/1.0; +http://example.com)"

    opts.CrawlDelay = 1 * time.Second
    opts.LogFlags = gocrawl.LogAll

    // Play nice with ddgo when running the test!
    opts.MaxVisits = 2

    // Create crawler and start at root of duckduckgo
    c := gocrawl.NewCrawlerWithOptions(opts)
    c.Run("https://duckduckgo.com/")

    // Remove "x" before Output: to activate the example (will run on go test)

    // xOutput: voluntarily fail to see log output
}

API

Gocrawl can be described as a minimalist web crawler (hence the "slim" tag, at ~1000 sloc), providing the basic engine upon which to build a full-fledged indexing machine with caching, persistence and staleness detection logic, or to use as is for quick and easy crawling. Gocrawl itself does not attempt to detect staleness of a page, nor does it implement a caching mechanism. If an URL is enqueued to be processed, it will make a request to fetch it (provided it is allowed by robots.txt - hence the "polite" tag). And there is no prioritization among the URLs to process, it assumes that all enqueued URLs must be visited at some point, and that the order in which they are is unimportant.

However, it does provide plenty of hooks and customizations. Instead of trying to do everything and impose a way to do it, it offers ways to manipulate and adapt it to anyone's needs.

As usual, the complete godoc reference can be found [here][godoc].

Design rationale

The major use-case behind gocrawl is to crawl some web pages while respecting the constraints of robots.txt policies and while applying a good web citizen crawl delay between each request to a given host. Hence the following design decisions:

• Each host spawns its own worker (goroutine) : This makes sense since it must first read its robots.txt data, and only then proceed sequentially, one request at a time, with the specified delay between each fetch. There are no constraints inter-host, so each separate worker can crawl independently.
• The visitor function is called on the worker goroutine : Again, this is ok because the crawl delay is likely bigger than the time required to parse the document, so this processing will usually not penalize the performance.
• Edge cases with no crawl delay are supported, but not optimized : In the rare but possible event when a crawl with no delay is needed (e.g.: on your own servers, or with permission outside busy hours, etc.), gocrawl accepts a null (zero) delay, but doesn't provide optimizations. That is, there is no "special path" in the code where visitor functions are de-coupled from the worker, or where multiple workers can be launched concurrently on the same host. (In fact, if this case is your only use-case, I would recommend not to use a library at all - since there is little value in it -, and simply use Go's standard libs and fetch at will with as many goroutines as are necessary.)
• Extender interface provides the means to write a drop-in, fully encapsulated behaviour : An implementation of Extender can radically enhance the core library, with caching, persistence, different fetching strategies, etc. This is why the Extender.Start() method is somewhat redundant with the Crawler.Run() method, Run allows calling the crawler as a library, while Start makes it possible to encapsulate the logic required to select the seed URLs into the Extender. The same goes for Extender.End() and the return value of Run.

Although it could probably be used to crawl a massive amount of web pages (after all, this is fetch, visit, enqueue, repeat ad nauseam!), most realistic (and um... tested!) use-cases should be based on a well-known, well-defined limited bucket of seeds. Distributed crawling is your friend, should you need to move past this reasonable use.

Crawler

The Crawler type controls the whole execution. It spawns worker goroutines and manages the URL queue. There are two helper constructors:

• NewCrawler(Extender) : Creates a crawler with the specified Extender instance.
• NewCrawlerWithOptions(*Options) : Creates a crawler with a pre-initialized *Options instance.

The one and only public function is Run(seeds interface{}) error which take a seeds argument (the base URLs used to start crawling) that can be expressed a number of different ways. It ends when there are no more URLs waiting to be visited, or when the Options.MaxVisit number is reached. It returns an error, which is ErrMaxVisits if this setting is what caused the crawling to stop.

The various types that can be used to pass the seeds are the following (the same types apply for the empty interfaces in Extender.Start(interface{}) interface{}, Extender.Visit(*URLContext, *http.Response, *goquery.Document) (interface{}, bool) and in Extender.Visited(*URLContext, interface{}), as well as the type of the EnqueueChan field):