Code
Hub
Workspaces
Following
Trending
Connect
MCP
copy
Create free account
hub
/
github.com/JFanZhao/spider
/ functions
Functions
80 in github.com/JFanZhao/spider
⨍
Functions
80
◇
Types & classes
26
↓ 13 callers
Method
put
添加一条数据 @param tableName @param row @param columnFamily @param column @param data @throws IOException
src/main/java/cn/ivan/spider/utils/HBaseUtils.java:204
↓ 12 callers
Method
add
(String nextUrl)
src/main/java/cn/ivan/spider/reposity/Repository.java:9
↓ 7 callers
Method
poll
()
src/main/java/cn/ivan/spider/reposity/Repository.java:5
↓ 5 callers
Method
getText
获取指定标签的值 @param tagNode @param xPath @return
src/main/java/cn/ivan/spider/utils/HtmlUtils.java:39
↓ 4 callers
Method
addField
(String key,String value)
src/main/java/cn/ivan/spider/domain/Page.java:59
↓ 4 callers
Method
getUrl
()
src/main/java/cn/ivan/spider/domain/Page.java:62
↓ 4 callers
Method
start
()
src/main/java/cn/ivan/spider/Spider.java:60
↓ 2 callers
Method
addHigh
(String nextUrl)
src/main/java/cn/ivan/spider/reposity/Repository.java:7
↓ 2 callers
Method
addUrl
(String url)
src/main/java/cn/ivan/spider/domain/Page.java:38
↓ 2 callers
Method
download
(String url)
src/main/java/cn/ivan/spider/download/Downloadable.java:6
↓ 2 callers
Method
execute
(JobExecutionContext arg0)
src/main/java/cn/ivan/spider/UrlJob.java:17
↓ 2 callers
Method
getAttributeByAttr
获取指定标签指定属性的值 @param tagNode @param Xpath @param attr @return
src/main/java/cn/ivan/spider/utils/HtmlUtils.java:49
↓ 2 callers
Method
getContent
根据url获取页面内容 @param url @return
src/main/java/cn/ivan/spider/utils/PageUtils.java:29
↓ 2 callers
Method
getTagNodeByXpath
获取指定的标签node @param tagNode @param xPath @return
src/main/java/cn/ivan/spider/utils/HtmlUtils.java:19
↓ 2 callers
Method
getTopDomain
获取url的顶级域名 @param url @return
src/main/java/cn/ivan/spider/utils/DomainUtils.java:15
↓ 2 callers
Method
getUrls
()
src/main/java/cn/ivan/spider/domain/Page.java:41
↓ 2 callers
Method
getValues
()
src/main/java/cn/ivan/spider/domain/Page.java:56
↓ 2 callers
Method
process
(Page page)
src/main/java/cn/ivan/spider/process/Processable.java:6
↓ 2 callers
Method
setProcessable
(Processable processable)
src/main/java/cn/ivan/spider/Spider.java:145
↓ 2 callers
Method
setStoreable
(Storeable storeable)
src/main/java/cn/ivan/spider/Spider.java:112
↓ 2 callers
Method
setUrl
(String url)
src/main/java/cn/ivan/spider/domain/Page.java:66
↓ 2 callers
Method
sleep
(long millons)
src/main/java/cn/ivan/spider/utils/SleepUtils.java:9
↓ 2 callers
Method
start
让当前进程一直运行
src/main/java/cn/ivan/spider/SpiderWatcher.java:78
↓ 2 callers
Method
store
(Page page)
src/main/java/cn/ivan/spider/store/Storeable.java:6
↓ 1 callers
Method
addHigh
(String nextUrl)
src/main/java/cn/ivan/spider/reposity/RandomRedisRepository.java:31
↓ 1 callers
Method
addHigh
@Title: addHigh @Description: 向队列仓库中添加url,在添加之前先截取url中的顶级域名,当作key值 @param nextUrl @throws
src/main/java/cn/ivan/spider/reposity/RandomQueueRepository.java:48
↓ 1 callers
Method
check
配置检查
src/main/java/cn/ivan/spider/Spider.java:94
↓ 1 callers
Method
download
下载页面 @param url @return
src/main/java/cn/ivan/spider/Spider.java:125
↓ 1 callers
Method
getContent
()
src/main/java/cn/ivan/spider/domain/Page.java:70
↓ 1 callers
Method
getGoodId
()
src/main/java/cn/ivan/spider/domain/Page.java:50
↓ 1 callers
Method
lrange
模拟redis的lrange方法 @param key @param start @param end @return
src/main/java/cn/ivan/spider/utils/RedisUtils.java:33
↓ 1 callers
Method
process
解析页面 @param page
src/main/java/cn/ivan/spider/Spider.java:136
↓ 1 callers
Method
processProduct
解析商品详细信息 @param page @param rootNode
src/main/java/cn/ivan/spider/process/JDProcessable.java:50
↓ 1 callers
Method
setContent
(String content)
src/main/java/cn/ivan/spider/domain/Page.java:74
↓ 1 callers
Method
setDownloadable
(Downloadable downloadable)
src/main/java/cn/ivan/spider/Spider.java:115
↓ 1 callers
Method
setGoodId
(String goodId)
src/main/java/cn/ivan/spider/domain/Page.java:53
↓ 1 callers
Method
setRepository
(Repository repository)
src/main/java/cn/ivan/spider/Spider.java:152
↓ 1 callers
Method
setSeedUrl
(String url)
src/main/java/cn/ivan/spider/Spider.java:163
↓ 1 callers
Method
store
(Page page)
src/main/java/cn/ivan/spider/Spider.java:159
Method
HBaseUtils
构造函数 加载配置
src/main/java/cn/ivan/spider/utils/HBaseUtils.java:58
Method
RedisUtils
构造方法 初始化redis配置信息
src/main/java/cn/ivan/spider/utils/RedisUtils.java:18
Method
Spider
()
src/main/java/cn/ivan/spider/Spider.java:42
Method
SpiderWatcher
()
src/main/java/cn/ivan/spider/SpiderWatcher.java:20
Method
add
(String lowKey, String url)
src/main/java/cn/ivan/spider/utils/RedisUtils.java:40
Method
add
向低优先级队列中存入url
src/main/java/cn/ivan/spider/reposity/QueueRepository.java:41
Method
add
(String nextUrl)
src/main/java/cn/ivan/spider/reposity/RedisRepository.java:19
Method
add
(String nextUrl)
src/main/java/cn/ivan/spider/reposity/RandomRedisRepository.java:41
Method
add
(String nextUrl)
src/main/java/cn/ivan/spider/reposity/RandomQueueRepository.java:58
Method
addHigh
向高优先级队列中存入url
src/main/java/cn/ivan/spider/reposity/QueueRepository.java:35
Method
addHigh
(String nextUrl)
src/main/java/cn/ivan/spider/reposity/RedisRepository.java:15
Method
createTable
创建一张表 @param tableName @param column @throws Exception
src/main/java/cn/ivan/spider/utils/HBaseUtils.java:240
Method
deleteOneRecord
删除一条记录 @param tableName @param rowKey
src/main/java/cn/ivan/spider/utils/HBaseUtils.java:159
Method
deleteTable
删除表 @param tableName @throws Exception
src/main/java/cn/ivan/spider/utils/HBaseUtils.java:142
Method
download
(String url)
src/main/java/cn/ivan/spider/download/HttpClientDownable.java:8
Method
getALLData
获取表中所有数据 @param tableName @throws Exception
src/main/java/cn/ivan/spider/utils/HBaseUtils.java:176
Method
getALLTable
查询所有表名 @return @throws Exception
src/main/java/cn/ivan/spider/utils/HBaseUtils.java:219
Method
getDownloadable
()
src/main/java/cn/ivan/spider/Spider.java:155
Method
getProcessable
()
src/main/java/cn/ivan/spider/Spider.java:141
Method
getRepository
()
src/main/java/cn/ivan/spider/Spider.java:149
Method
getRowFilter
rowFilter 的使用 @param tableName @param reg @throws Exception
src/main/java/cn/ivan/spider/utils/HBaseUtils.java:105
Method
getScanData
扫描数据 @param tableName @param family @param qualifier @throws Exception
src/main/java/cn/ivan/spider/utils/HBaseUtils.java:122
Method
getStoreable
()
src/main/java/cn/ivan/spider/Spider.java:108
Method
main
(String[] args)
src/main/java/cn/ivan/spider/Spider.java:166
Method
main
(String[] args)
src/main/java/cn/ivan/spider/UrlManager.java:17
Method
main
(String[] args)
src/main/java/cn/ivan/spider/SpiderWatcher.java:70
Method
main
(String[] args)
src/main/java/cn/ivan/spider/utils/HBaseUtils.java:77
Method
poll
(String key)
src/main/java/cn/ivan/spider/utils/RedisUtils.java:47
Method
poll
从url仓库中取url,先从高优先级队列中取,如果高优先级队列中没有url,则从低优先级队列中取
src/main/java/cn/ivan/spider/reposity/QueueRepository.java:25
Method
poll
()
src/main/java/cn/ivan/spider/reposity/RedisRepository.java:7
Method
poll
()
src/main/java/cn/ivan/spider/reposity/RandomRedisRepository.java:23
Method
poll
@Title: poll @Description: 随机获取一个url @return @throws
src/main/java/cn/ivan/spider/reposity/RandomQueueRepository.java:34
Method
process
当监视器发现监控的父节点下面的子节点发生变化的时候,会调用这个方法 注意:watcher监视器是单次有效的,注册一次,只能使用一次,需要想要一直使用,则需要重复注册
src/main/java/cn/ivan/spider/SpiderWatcher.java:41
Method
process
(Page page)
src/main/java/cn/ivan/spider/process/JDProcessable.java:18
Method
run
()
src/main/java/cn/ivan/spider/Spider.java:67
Method
setUrls
(List<String> urls)
src/main/java/cn/ivan/spider/domain/Page.java:44
Method
setValues
(Map<String, String> values)
src/main/java/cn/ivan/spider/domain/Page.java:47
Method
store
(Page page)
src/main/java/cn/ivan/spider/store/ConsoleStoreable.java:7
Method
store
(Page page)
src/main/java/cn/ivan/spider/store/HBaseStoreable.java:12
Method
test
()
src/test/java/cn/ivan/spider/CuratorTest.java:13
Method
testName
()
src/test/java/cn/ivan/spider/SpiderTest.java:12