Types & classes — github.com/JFanZhao/spider

CodeHub MCPcopy Create free account

hub / github.com/JFanZhao/spider / types & classes

Types & classes26 in github.com/JFanZhao/spider

⨍ Functions 80 ◇ Types & classes 26

把项目中可能会发生变化的参数的值都提取到这个类中，注意:在这我直接把值在这个类中写死了，实际中这些参数的值是需要读取配置文件或者从数据库读取的如果读取配置文件，可以使用properties类 @author Administrator

src/main/java/cn/ivan/spider/utils/Config.java:11

ClassConsoleStoreable

src/main/java/cn/ivan/spider/store/ConsoleStoreable.java:5

ClassCuratorTest

src/test/java/cn/ivan/spider/CuratorTest.java:12

ClassDomainUtils

src/main/java/cn/ivan/spider/utils/DomainUtils.java:8

InterfaceDownloadable

src/main/java/cn/ivan/spider/download/Downloadable.java:5

ClassHBaseStoreable

src/main/java/cn/ivan/spider/store/HBaseStoreable.java:9

ClassHBaseUtils

src/main/java/cn/ivan/spider/utils/HBaseUtils.java:28

html页面的utils类 @author Hades

src/main/java/cn/ivan/spider/utils/HtmlUtils.java:12

ClassHttpClientDownable

src/main/java/cn/ivan/spider/download/HttpClientDownable.java:6

ClassJDProcessable

src/main/java/cn/ivan/spider/process/JDProcessable.java:16

存储页面的原始数据 @author Hades

src/main/java/cn/ivan/spider/domain/Page.java:13

页面帮助类 @author Hades

src/main/java/cn/ivan/spider/utils/PageUtils.java:22

InterfaceProcessable

src/main/java/cn/ivan/spider/process/Processable.java:5

ClassQueueRepository

@author ivan Email:seeHades@163.com @date 2016年1月7日上午12:45:11 @desc 优先级队列url 仓库

src/main/java/cn/ivan/spider/reposity/QueueRepository.java:13

ClassRandomQueueRepository

@author ivan Email:seeHades@163.com @date 2016年1月7日上午12:49:20 @desc 使用对列随机获取某一个电商网站的url

src/main/java/cn/ivan/spider/reposity/RandomQueueRepository.java:20

ClassRandomRedisRepository

随机获取某一个电商网站的url @author Hades

src/main/java/cn/ivan/spider/reposity/RandomRedisRepository.java:19

ClassRedisRepository

src/main/java/cn/ivan/spider/reposity/RedisRepository.java:5

ClassRedisUtils

src/main/java/cn/ivan/spider/utils/RedisUtils.java:9

InterfaceRepository

src/main/java/cn/ivan/spider/reposity/Repository.java:3

ClassSleepUtils

src/main/java/cn/ivan/spider/utils/SleepUtils.java:6

src/main/java/cn/ivan/spider/Spider.java:32

ClassSpiderTest

src/test/java/cn/ivan/spider/SpiderTest.java:11

ClassSpiderWatcher

Spider 使用Curator监控爬虫节点的生命周期 @author Hades

src/main/java/cn/ivan/spider/SpiderWatcher.java:17

InterfaceStoreable

src/main/java/cn/ivan/spider/store/Storeable.java:5

将种子url 添加到redis的队列中 @author Hades

src/main/java/cn/ivan/spider/UrlJob.java:15

ClassUrlManager

url调度类负责每天定时向url仓库中添加种子url 这样可以保证爬虫每天都抓取一次所有数据 @author Administrator

src/main/java/cn/ivan/spider/UrlManager.java:16