每个爬取任务的数据容器

The Downloader interface. You can implement the interface by implement function Download. Function Download need to return Page instance pointer that

The HttpDownloader download response by package net/http. The "html" content is contained in dom parser of package goquery. The "json" content is save

GUI输入

GUI内容 下拉菜单辅助结构体

Request represents object waiting for being crawled.

Response represents an entity be crawled.

采集规则单元

采集规则树

SrcManage is an interface that who want implement an management object can realize these functions.

Filelog represents an active object that logs on file to record error or other useful info. The filelog info is output to os.Stderr. The loginst is an

The plog is a public function combiation for other log objects.

Strace represents an active object that strace the processing of spider. The strace info is output to os.Stderr. The loginst is an point of logger in