gathers links to be visited in the future from a web page's body. does it by finding "href" attributes in the DOM :param body

Removes a link from our current links list and blacklists it so we don't visit it in the future :param link: link to remove a

Sends a POST/GET requests using a random user agent :param url: the url to visit :return: the response Requests object

Selects a random link out of the available link list and visits it. Blacklists any link that is not responsive or that contains no ot

Checks is a URL is blacklisted :param url: full URL :return: boolean indicating whether a URL is blacklisted or not

Determines whether the specified timeout has reached, if no timeout is specified then return false :return: boolean indicatin

Check if a url is a valid url. Used to filter out invalid values that were found in the "href" attribute, for example "javasc

Normalizes links extracted from the DOM by making them all absolute, so we can request them, for example, turns a "/images" link extr

Collects links from our root urls, stores them and then calls `_browse_from_links` to browse them

Loads and decodes a JSON config file, sets the config of the crawler instance to the loaded one :param file_path: path of the

Sets the config of the crawler instance to the provided dict :param config: dict of configuration options, for example: {

Sets a specific key in the config dict :param option: the option key in the config, for example: "max_depth" :param value: va

Initializes the Crawl class

filters url if it is blacklisted or not valid, we put filtering logic here :param url: full url to be checked :return: boolea