Types & classes — github.com/CrawlScript/WebCollector

ClassAbuyunDynamicProxyRequester

WebCollector使用阿布云代理的Http请求插件 <p> 插件使用方法： crawler.setRequester(new AbuyunDynamicProxyRequester("阿布云动态代理用户名", "阿布云动态代理密码")); 一行代码解决，无需自定义代理池及其它代理切换规则 <p

src/main/java/cn/edu/hfut/dmic/webcollector/example/AbuyunDynamicProxyRequester.java:39

ClassAutoParseCrawler

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/crawler/AutoParseCrawler.java:39

ClassBerkeleyCrawler

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/berkeley/BerkeleyCrawler.java:27

ClassBerkeleyDBManager

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/berkeley/BerkeleyDBManager.java:43

ClassBerkeleyDBReader

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/berkeley/BerkeleyDBReader.java:37

ClassBerkeleyDBUtils

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/berkeley/BerkeleyDBUtils.java:33

ClassBerkeleyGenerator

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/berkeley/BerkeleyGenerator.java:40

ClassBreadthCrawler

cn.edu.hfut.dmic.webcollector.plugin.rocks.BreadthCrawler是基于RocksDB的插件,于2.72版重新设计 BreadthCrawler可以设置正则规律，让遍历器自动根据URL的正则遍历网站，可以关闭这个功能，自定义遍历如果autoParse

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/BreadthCrawler.java:32

ClassBreadthCrawler

cn.edu.hfut.dmic.webcollector.plugin.berkeley.BreadthCrawler是基于Berkeley DB的插件,于2.20版重新设计 BreadthCrawler可以设置正则规律，让遍历器自动根据URL的正则遍历网站，可以关闭这个功能，自定义遍历如果au

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/berkeley/BreadthCrawler.java:32

ClassCharsetDetector

字符集自动检测 @author hu

src/main/java/cn/edu/hfut/dmic/webcollector/util/CharsetDetector.java:30

ClassCharsetDetectorTest

Created by Vlad Medvedev on 21.01.2016. vladislav.medvedev@devfactory.com

src/test/java/cn/edu/hfut/dmic/webcollector/util/CharsetDetectorTest.java:31

ClassCommonConfigured

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/conf/CommonConfigured.java:23

ClassCommonRequester

src/main/java/cn/edu/hfut/dmic/webcollector/net/CommonRequester.java:7

ClassConfig

全局配置 @author hu

src/main/java/cn/edu/hfut/dmic/webcollector/util/Config.java:25

ClassConfiguration

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/conf/Configuration.java:28

ClassConfigurationUtils

src/main/java/cn/edu/hfut/dmic/webcollector/util/ConfigurationUtils.java:6

InterfaceConfigured

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/conf/Configured.java:23

ClassContentExtractor

ContentExtractor could extract content,title,time from news webpage @author hu

src/main/java/cn/edu/hfut/dmic/contentextractor/ContentExtractor.java:43

ClassCountInfo

src/main/java/cn/edu/hfut/dmic/contentextractor/ContentExtractor.java:55

ClassCounter

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/util/Counter.java:26

ClassCrawlDatum

爬取任务的数据结构 @author hu

src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatum.java:33

ClassCrawlDatumFormater

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/util/CrawlDatumFormater.java:35

ClassCrawlDatumTest

@author hu

src/test/java/cn/edu/hfut/dmic/webcollector/util/CrawlDatumTest.java:30

ClassCrawlDatums

用于存储多个CrawlDatum的数据结构 @author hu

src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatums.java:31

ClassCrawlDatumsTest

@author hu

src/test/java/cn/edu/hfut/dmic/webcollector/util/CrawlDatumsTest.java:31

ClassCrawler

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/crawler/Crawler.java:37

ClassDBManager

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/crawldb/DBManager.java:30

ClassDBManagerTest

src/test/java/cn/edu/hfut/dmic/webcollector/util/DBManagerTest.java:16

ClassDefaultConfigured

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/conf/DefaultConfigured.java:23

ClassDemoAnnotatedAutoNewsCrawler

Crawling news from github news @author hu

src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoAnnotatedAutoNewsCrawler.java:12

ClassDemoAnnotatedBingCrawler

本教程演示了WebCollector 2.20的新特性: 1)MetaData: MetaData是每个爬取任务的附加信息,灵活应用MetaData可以大大简化爬虫的设计. 例如Post请求往往需要包含参数，而传统爬虫单纯使用URL来保存参数的方法不适合复杂的POST请求. 一些

src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoAnnotatedBingCrawler.java:52

ClassDemoAnnotatedDepthCrawler

本教程和深度遍历没有任何关系一些爬取需求希望加入深度信息，即遍历树中网页的层利用2.20版本中的新特性MetaData可以轻松实现这个功能 @author hu

src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoAnnotatedDepthCrawler.java:32

ClassDemoAnnotatedManualNewsCrawler

Crawling news from github news @author hu

src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoAnnotatedManualNewsCrawler.java:12

ClassDemoAnnotatedMatchTypeCrawler

WebCollector 2.x版本的tutorial(2.20以上) 2.x版本特性： 1）自定义遍历策略，可完成更为复杂的遍历业务，例如分页、AJAX 2）可以为每个URL设置附加信息(MetaData)，利用附加信息可以完成很多复杂业务，例如深度获取、锚文本获取、引用页面获取、POST参数传递

src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoAnnotatedMatchTypeCrawler.java:42

ClassDemoAnnotatedRedirectCrawler

Handle 301/302 redirection @author hu

src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoAnnotatedRedirectCrawler.java:33

ClassDemoAutoNewsCrawler

Crawling news from github news @author hu

src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoAutoNewsCrawler.java:12

ClassDemoBingCrawler

本教程演示了WebCollector 2.20的新特性: 1)MetaData: MetaData是每个爬取任务的附加信息,灵活应用MetaData可以大大简化爬虫的设计. 例如Post请求往往需要包含参数，而传统爬虫单纯使用URL来保存参数的方法不适合复杂的POST请求. 一些

src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoBingCrawler.java:53

ClassDemoCookieCrawler

教程：使用WebCollector自定义Http请求可以自定义User-Agent和Cookie @author hu

src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoCookieCrawler.java:16

ClassDemoExceptionCrawler

Crawling news from github news @author hu

src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoExceptionCrawler.java:13

ClassDemoHashSetNextFilter

WebCollector 2.x版本的tutorial(2.20以上) 2.x版本特性： 1）自定义遍历策略，可完成更为复杂的遍历业务，例如分页、AJAX 2）可以为每个URL设置附加信息(MetaData)，利用附加信息可以完成很多复杂业务，例如深度获取、锚文本获取、引用页面获取、POST参数传递

src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoHashSetNextFilter.java:39

ClassDemoManualNewsCrawler

Crawling news from github news @author hu

src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoManualNewsCrawler.java:12

ClassDemoMetaCrawler

WebCollector 2.x版本的tutorial(2.20以上) 2.x版本特性： 1）自定义遍历策略，可完成更为复杂的遍历业务，例如分页、AJAX 2）可以为每个URL设置附加信息(MetaData)，利用附加信息可以完成很多复杂业务，例如深度获取、锚文本获取、引用页面获取、POST参数传递

src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoMetaCrawler.java:42

ClassDemoNextFilter

WebCollector 2.x版本的tutorial(2.20以上) 2.x版本特性： 1）自定义遍历策略，可完成更为复杂的遍历业务，例如分页、AJAX 2）可以为每个URL设置附加信息(MetaData)，利用附加信息可以完成很多复杂业务，例如深度获取、锚文本获取、引用页面获取、POST参数传递

src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoNextFilter.java:39

ClassDemoPostCrawler

本教程演示了如何自定义http请求有些爬取任务中，可能只有部分URL需要使用POST请求，我们可以利用2.20版本中添加的MetaData功能，来完成POST请求的定制。使用MetaData除了可以标记URL是否需要使用POST，还可以存储POST所需的参数信息教程中还演示了如何定制C

src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoPostCrawler.java:45

ClassDemoRandomProxyCrawler

教程：WebCollector随机代理 @author hu

src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoRandomProxyCrawler.java:24

ClassDemoRedirectCrawler

Handle 301/302 redirection @author hu

src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoRedirectCrawler.java:36

ClassDemoSeleniumCrawler

本教程演示如何利用WebCollector爬取javascript生成的数据 @author hu

src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoSeleniumCrawler.java:37

ClassDemoTypeCrawler

WebCollector 2.40新特性 page.matchType 在添加CrawlDatum时（添加种子、或在抓取时向next中添加任务），可以为CrawlDatum设置type信息 type的本质也是meta信息，为CrawlDatum的附加信息在添加种子或向next中添加任务时，设置

src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoTypeCrawler.java:45

ClassExceptionUtils

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/util/ExceptionUtils.java:23

InterfaceExecutor

Created by hu on 2016/2/18.

src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/Executor.java:26

ClassFetchItem

src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/Fetcher.java:81

ClassFetchQueue

src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/Fetcher.java:90

ClassFetcher

抓取器 @author hu

src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/Fetcher.java:42

ClassFetcherThread

src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/Fetcher.java:207

ClassFileIdGenerator

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/util/FileIdGenerator.java:26

ClassFileSystemOutput

FileSystemOutput并不属于WebCollector内核，它只是实现一个简单的输出，将网页根据url路径，保存到本地目录，按照网站目录结构来存储网站内容。BreadthCrawler的visit函数中，默认使用 FileSystemOutput来保存网页。不推荐使用FileSyste

src/main/java/cn/edu/hfut/dmic/webcollector/util/FileSystemOutput.java:33

ClassFileUtils

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/util/FileUtils.java:31

ClassGenerator

抓取任务生成器 @author hu

src/main/java/cn/edu/hfut/dmic/webcollector/crawldb/Generator.java:32

InterfaceGeneratorFilter

src/main/java/cn/edu/hfut/dmic/webcollector/crawldb/GeneratorFilter.java:6

ClassGsonUtils

src/main/java/cn/edu/hfut/dmic/webcollector/util/GsonUtils.java:6

ClassHashSetNextFilter

Filter nextItem whose key is contained in the hashset @author hu

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/nextfilter/HashSetNextFilter.java:28

InterfaceInjector

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/crawldb/Injector.java:26

ClassInvalidAnnotatedVisitorMethodException

src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/Visitor.java:86

ClassJsoupUtils

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/util/JsoupUtils.java:30

ClassLinks

用于存储多个URL的数据结构，继承于ArrayList @author hu

src/main/java/cn/edu/hfut/dmic/webcollector/model/Links.java:34

ClassListUtils

src/main/java/cn/edu/hfut/dmic/webcollector/util/ListUtils.java:6

ClassMD5Utils

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/util/MD5Utils.java:30

InterfaceMetaGetter

src/main/java/cn/edu/hfut/dmic/webcollector/model/MetaGetter.java:5

InterfaceMetaSetter

src/main/java/cn/edu/hfut/dmic/webcollector/model/MetaSetter.java:5

ClassMetaTest

src/test/java/cn/edu/hfut/dmic/webcollector/util/MetaTest.java:11

ClassMyRequester

src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoCookieCrawler.java:20

ClassMyRequester

src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoRandomProxyCrawler.java:28

ClassMysqlHelper

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/util/MysqlHelper.java:31

ClassNews

@author hu

src/main/java/cn/edu/hfut/dmic/contentextractor/News.java:26

InterfaceNextFilter

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/NextFilter.java:26

ClassOkHttpRequester

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/net/OkHttpRequester.java:34

ClassOkHttpRequesterTest

src/test/java/cn/edu/hfut/dmic/webcollector/util/OkHttpRequesterTest.java:9

ClassPage

Page是爬取过程中，内存中保存网页爬取信息的一个容器，Page只在内存中存放，用于保存一些网页信息，方便用户进行自定义网页解析之类的操作。 @author hu

src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:43

ClassProxies

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/net/Proxies.java:37

ClassQueueFeeder

src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/Fetcher.java:129

ClassRamCrawler

基于内存的Crawler插件，适合一次性爬取，并不具有断点爬取功能长期任务请使用BreadthCrawler @author hu

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/ram/RamCrawler.java:28

ClassRamDB

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/ram/RamDB.java:30

ClassRamDBManager

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/ram/RamDBManager.java:32

ClassRamGenerator

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/ram/RamGenerator.java:30

ClassReflectionUtils

src/main/java/cn/edu/hfut/dmic/webcollector/util/ReflectionUtils.java:6

ClassRegexRule

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/util/RegexRule.java:29

InterfaceRequester

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/net/Requester.java:28

ClassRocksCrawler

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksCrawler.java:27

ClassRocksDBManager

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksDBManager.java:40

ClassRocksDBReader

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksDBReader.java:33

ClassRocksDBUtils

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksDBUtils.java:37

ClassRocksGenerator

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksGenerator.java:31

InterfaceSegmentWriter

爬取过程中，写入爬取历史、网页Content、解析信息的Writer @author hu

src/main/java/cn/edu/hfut/dmic/webcollector/crawldb/SegmentWriter.java:28

ClassStatusGeneratorFilter

src/main/java/cn/edu/hfut/dmic/webcollector/crawldb/StatusGeneratorFilter.java:6

ClassTutorialCrawler

WebCollector 2.x版本的tutorial(2.20以上) 2.x版本特性： 1）自定义遍历策略，可完成更为复杂的遍历业务，例如分页、AJAX 2）可以为每个URL设置附加信息(MetaData)，利用附加信息可以完成很多复杂业务，例如深度获取、锚文本获取、引用页面获取、POST参数传递

src/main/java/cn/edu/hfut/dmic/webcollector/example/TutorialCrawler.java:42

InterfaceVisitor

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/Visitor.java:34

ClassVisitorMethodDispatcher

src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/VisitorMethodDispatcher.java:20

Types & classes97 in github.com/CrawlScript/WebCollector