Control how we handle robots.txt

Types of links

Run mode specifies a couple of different ways for handling general operations.

CSVResultsManager writes a CSV containing all of the results.

Client is a thin wrapper around http.Client to make enhancements to support our use case.

A ClientFactory allows constructing HTTP Clients based on various Dialers or Transports.

DurationFlag is a flag.Value that takes a Duration spec (see time.Duration) and parses it and stores the Duration.

An Expander is responsible for taking input URLs and expanding them to the various mutations to be processed.

Expand extensions when none are given

HTMLResultsManager writes an HTML file containing the results.

Header expander tries each of the headers known to it in turn

HeaderFlag is an http.Header wrapped in the flag.Value interface.

IntSliceFlag is a flag.Value that takes a comma-separated string and turns it into a slice of ints.

Check results for broken links.

PlainResultsManager is designed to output a very basic output that is good for human reading, but not so good for machine parsing. This is the defaul

ProxyClientFactory uses the h12.io/socks package to support SOCKS proxies when transporting requests to the webserver.

This is the result emitted by the worker for each URL tested.

ResultsManager provides an interface for reading results from a channel and writing them to some form of output.

ScanSettings store all of the settings for the running scan. It's basically a mapping from command-line flags into a single struct that can be passed

StringSliceFileFlag is flag.Value that loads from a file into a wrapped StringSliceFlag

StringSliceFlag is a flag.Value that takes a comma-separated or repeated string and turns it into a slice of strings.

A WordMangler is responsible for modifying a wordlist entry to produce alternatives.

An Expander is responsible for taking input URLs and expanding them to include all of the words in the wordlist.

Count work to do and work done

WorkFilter is responsible for making sure that a given URL is only tested once, and also for applying any exclusion rules to prevent URLs from being s

WorkQueue is a singleton that maintains the queue of work to be done. It reads from one input channel, verifies that the URL is in scope, queues it, t

Workers do the work of connecting to the server, issuing the request, and then optionally parsing the response. Normally a pool of several workers wi

httpClient is the concrete implementation of the Client interface. The Client member is almost always an http.Client outside of tests.

This interface just allows us to substitute a mock in tests

Write link check output in CSV

Write link check output in HTML

Mock httpClient that checks auth with "user" and "pass"

Mock httpClient that returns arbitrary responses