Code
Hub
Workspaces
Following
Trending
Connect
MCP
copy
Create free account
hub
/
github.com/CrawlScript/WebCollector
/ functions
Functions
510 in github.com/CrawlScript/WebCollector
⨍
Functions
510
◇
Types & classes
97
↓ 52 callers
Method
meta
(JsonObject metaData)
src/main/java/cn/edu/hfut/dmic/webcollector/model/MetaSetter.java:6
↓ 50 callers
Method
add
(String url)
src/main/java/cn/edu/hfut/dmic/webcollector/model/Links.java:54
↓ 33 callers
Method
get
(int index)
src/main/java/cn/edu/hfut/dmic/webcollector/model/Links.java:167
↓ 28 callers
Method
getConf
()
src/main/java/cn/edu/hfut/dmic/webcollector/conf/Configured.java:24
↓ 23 callers
Method
set
(String key, Object value)
src/main/java/cn/edu/hfut/dmic/webcollector/conf/Configuration.java:53
↓ 22 callers
Method
size
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/Links.java:171
↓ 22 callers
Method
start
开始爬取,迭代次数为depth @param depth 迭代次数 @throws Exception 异常
src/main/java/cn/edu/hfut/dmic/webcollector/crawler/Crawler.java:91
↓ 22 callers
Method
url
返回网页的url @return 网页的url
src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:353
↓ 20 callers
Method
type
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatum.java:128
↓ 18 callers
Method
addSeedAndReturn
(CrawlDatum datum, boolean force)
src/main/java/cn/edu/hfut/dmic/webcollector/crawler/Crawler.java:288
↓ 18 callers
Method
get
(String key)
src/main/java/cn/edu/hfut/dmic/webcollector/conf/Configuration.java:73
↓ 17 callers
Method
addSeed
添加种子任务 @param datum 种子任务 @param force 如果添加的种子是已爬取的任务,当force为true时,会强制注入种子,当force为false时,会忽略该种子
src/main/java/cn/edu/hfut/dmic/webcollector/crawler/Crawler.java:169
↓ 17 callers
Method
key
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:505
↓ 17 callers
Method
put
(RocksDB rocksDB, String key,String value)
src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksDBUtils.java:103
↓ 16 callers
Method
select
(URI uri)
src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoRandomProxyCrawler.java:45
↓ 15 callers
Method
parse
(String json)
src/main/java/cn/edu/hfut/dmic/webcollector/util/GsonUtils.java:10
↓ 13 callers
Method
addAndReturn
(String url)
src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatums.java:90
↓ 12 callers
Method
add
(CrawlDatum datum)
src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatums.java:56
↓ 12 callers
Method
encode
(String charset, String text)
src/test/java/cn/edu/hfut/dmic/webcollector/util/CharsetDetectorTest.java:51
↓ 12 callers
Method
toString
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/Links.java:196
↓ 11 callers
Method
close
()
src/main/java/cn/edu/hfut/dmic/webcollector/crawldb/DBManager.java:47
↓ 11 callers
Method
meta
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatum.java:189
↓ 11 callers
Method
select
(String cssSelector)
src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:274
↓ 10 callers
Method
addRegex
添加URL正则约束 @param urlRegex URL正则约束
src/main/java/cn/edu/hfut/dmic/webcollector/crawler/AutoParseCrawler.java:100
↓ 10 callers
Method
close
()
src/main/java/cn/edu/hfut/dmic/webcollector/plugin/berkeley/BerkeleyDBManager.java:125
↓ 10 callers
Method
doc
返回网页解析后的DOM树(Jsoup的Document对象) 已废弃,使用doc()方法代替 @return 网页解析后的DOM树
src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:403
↓ 10 callers
Method
html
返回网页的源码字符串 @return 网页的源码字符串
src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:363
↓ 10 callers
Method
metaAsInt
(String key)
src/main/java/cn/edu/hfut/dmic/webcollector/model/MetaGetter.java:8
↓ 10 callers
Method
setThreads
设置爬虫的线程数 @param threads 爬虫的线程数
src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/Fetcher.java:413
↓ 9 callers
Method
attr
获取网页中满足指定css选择器的所有元素的指定属性的集合 例如通过attr("img[src]","abs:src")可获取网页中第一个图片的链接 @param cssSelector @param attrName @return
src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:157
↓ 9 callers
Method
close
()
src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksDBManager.java:110
↓ 9 callers
Method
code
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:510
↓ 9 callers
Method
get
(RocksDB rocksDB, String key)
src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksDBUtils.java:107
↓ 8 callers
Method
links
(boolean parseImg)
src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:162
↓ 8 callers
Method
matchType
判断当前Page(CrawlDatum)的type是否为type @param type @return 是否相等
src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:88
↓ 8 callers
Method
set
(int value)
src/main/java/cn/edu/hfut/dmic/webcollector/util/Counter.java:44
↓ 7 callers
Method
checkMethod
(Method method)
src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/VisitorMethodDispatcher.java:36
↓ 7 callers
Method
fail
(String message)
src/main/java/cn/edu/hfut/dmic/webcollector/util/ExceptionUtils.java:24
↓ 7 callers
Method
isEmpty
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/Links.java:187
↓ 7 callers
Method
selectText
(String cssSelector, int index)
src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:215
↓ 7 callers
Method
url
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatum.java:154
↓ 6 callers
Method
asJsonArray
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatum.java:299
↓ 6 callers
Method
getResponse
(String url)
src/main/java/cn/edu/hfut/dmic/webcollector/net/Requester.java:30
↓ 6 callers
Method
guessEncoding
根据字节数组,猜测可能的字符集,如果检测失败,返回utf-8 @param content 待检测的字节数组 @return 可能的字符集,如果检测失败,返回utf-8
src/main/java/cn/edu/hfut/dmic/webcollector/util/CharsetDetector.java:110
↓ 6 callers
Method
guessEncodingByMozilla
根据字节数组,猜测可能的字符集,如果检测失败,返回utf-8 @param bytes 待检测的字节数组 @return 可能的字符集,如果检测失败,返回utf-8
src/main/java/cn/edu/hfut/dmic/webcollector/util/CharsetDetector.java:92
↓ 6 callers
Method
location
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:514
↓ 6 callers
Method
setTo
(Configured from, Object... targets)
src/main/java/cn/edu/hfut/dmic/webcollector/util/ConfigurationUtils.java:14
↓ 6 callers
Method
strToEntry
(String str)
src/main/java/cn/edu/hfut/dmic/webcollector/plugin/berkeley/BerkeleyDBUtils.java:56
↓ 5 callers
Method
addRule
添加一个正则规则 正则规则有两种,正正则和反正则 URL符合正则规则需要满足下面条件: 1.至少能匹配一条正正则 2.不能和任何反正则匹配 正正则示例:+a. c是一条正正则,正则的内容为a. c,起始加号表示正正则 反正则示例:-a. c时一条反正则,正则的内容为a. c,起始减号表示反正则 如果
src/main/java/cn/edu/hfut/dmic/webcollector/util/RegexRule.java:67
↓ 5 callers
Method
content
返回网页/文件的内容 @return 网页/文件的内容
src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:339
↓ 5 callers
Method
copyMeta
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/MetaGetter.java:12
↓ 5 callers
Method
key
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatum.java:239
↓ 5 callers
Method
next
()
src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksDBReader.java:51
↓ 5 callers
Method
openCrawldbDatabase
(String crawlPath)
src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksDBUtils.java:59
↓ 5 callers
Method
put
(Database database,String key,String value)
src/main/java/cn/edu/hfut/dmic/webcollector/plugin/berkeley/BerkeleyDBUtils.java:52
↓ 5 callers
Method
setTopN
(Integer topN)
src/main/java/cn/edu/hfut/dmic/webcollector/conf/Configuration.java:105
↓ 4 callers
Method
contentType
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:394
↓ 4 callers
Method
getContentElementByDoc
(Document doc)
src/main/java/cn/edu/hfut/dmic/contentextractor/ContentExtractor.java:436
↓ 4 callers
Method
getSize
()
src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/Fetcher.java:100
↓ 4 callers
Method
getStatus
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatum.java:180
↓ 4 callers
Method
matchUrl
判断当前Page的URL是否和输入正则匹配 @param urlRegex @return
src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:69
↓ 4 callers
Method
open
(String dbPath)
src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksDBUtils.java:87
↓ 4 callers
Method
regex
(String regex, int group, String defaultResult)
src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:287
↓ 4 callers
Method
remove
(int index)
src/main/java/cn/edu/hfut/dmic/webcollector/model/Links.java:175
↓ 4 callers
Method
satisfy
判断输入字符串是否符合正则规则 @param str 输入的字符串 @return 输入字符串是否符合正则规则
src/main/java/cn/edu/hfut/dmic/webcollector/util/RegexRule.java:112
↓ 4 callers
Method
selectTextList
(String cssSelector)
src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:206
↓ 4 callers
Method
setExecuteInterval
(Integer executeInterval)
src/main/java/cn/edu/hfut/dmic/webcollector/conf/Configuration.java:133
↓ 4 callers
Method
write
(String fileName, String contentStr, String charset)
src/main/java/cn/edu/hfut/dmic/webcollector/util/FileUtils.java:60
↓ 3 callers
Method
addByRegex
(Element ele, RegexRule regexRule, boolean parseSrc)
src/main/java/cn/edu/hfut/dmic/webcollector/model/Links.java:134
↓ 3 callers
Method
createCrawlDatum
(byte[] key, byte[] value)
src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksDBUtils.java:124
↓ 3 callers
Method
createCrawlDatum
(DatabaseEntry key,DatabaseEntry value)
src/main/java/cn/edu/hfut/dmic/webcollector/plugin/berkeley/BerkeleyDBUtils.java:60
↓ 3 callers
Method
createDefaultDBOptions
()
src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksDBUtils.java:83
↓ 3 callers
Method
datumToString
(CrawlDatum datum)
src/main/java/cn/edu/hfut/dmic/webcollector/util/CrawlDatumFormater.java:39
↓ 3 callers
Method
getContentElement
()
src/main/java/cn/edu/hfut/dmic/contentextractor/ContentExtractor.java:149
↓ 3 callers
Method
getExecuteCount
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatum.java:172
↓ 3 callers
Method
inject
(CrawlDatum datum)
src/main/java/cn/edu/hfut/dmic/webcollector/crawldb/Injector.java:27
↓ 3 callers
Method
next
return null if there is no CrawlDatum to generate @return
src/main/java/cn/edu/hfut/dmic/webcollector/crawldb/Generator.java:49
↓ 3 callers
Method
setRequester
(Requester requester)
src/main/java/cn/edu/hfut/dmic/webcollector/crawler/AutoParseCrawler.java:161
↓ 3 callers
Method
setStatus
(int status)
src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatum.java:184
↓ 3 callers
Method
size
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatums.java:180
↓ 3 callers
Method
strToKeyOrValue
(String str)
src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksDBUtils.java:120
↓ 2 callers
Method
addPositive
添加一个正正则规则 @param positiveregex @return 自身
src/main/java/cn/edu/hfut/dmic/webcollector/util/RegexRule.java:90
↓ 2 callers
Method
briefInfo
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatum.java:225
↓ 2 callers
Method
charset
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:493
↓ 2 callers
Method
clear
()
src/main/java/cn/edu/hfut/dmic/webcollector/plugin/ram/RamDBManager.java:49
↓ 2 callers
Method
createGenerator
()
src/main/java/cn/edu/hfut/dmic/webcollector/crawldb/DBManager.java:36
↓ 2 callers
Method
deleteDir
(File dir)
src/main/java/cn/edu/hfut/dmic/webcollector/util/FileUtils.java:33
↓ 2 callers
Method
execute
(CrawlDatum datum,CrawlDatums next)
src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/Executor.java:27
↓ 2 callers
Method
fromJsonArray
(String crawlDatumKey, JsonArray jsonArray)
src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatum.java:315
↓ 2 callers
Method
get
(int index)
src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatums.java:176
↓ 2 callers
Method
get
()
src/main/java/cn/edu/hfut/dmic/webcollector/util/Counter.java:48
↓ 2 callers
Method
getContent
()
src/main/java/cn/edu/hfut/dmic/contentextractor/News.java:51
↓ 2 callers
Method
getData
()
src/main/java/cn/edu/hfut/dmic/webcollector/conf/Configuration.java:86
↓ 2 callers
Method
getDefault
()
src/main/java/cn/edu/hfut/dmic/webcollector/conf/Configuration.java:194
↓ 2 callers
Method
getExecuteTime
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatum.java:164
↓ 2 callers
Method
getFetchPath
(String crawlPath)
src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksDBUtils.java:44
↓ 2 callers
Method
getFullMethodName
(Method method)
src/main/java/cn/edu/hfut/dmic/webcollector/util/ReflectionUtils.java:41
↓ 2 callers
Method
getInteger
(String key)
src/main/java/cn/edu/hfut/dmic/webcollector/conf/Configuration.java:62
↓ 2 callers
Method
getLinkPath
(String crawlPath)
src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksDBUtils.java:49
↓ 2 callers
Method
getNewsByDoc
(Document doc)
src/main/java/cn/edu/hfut/dmic/contentextractor/ContentExtractor.java:489
next →
1–100 of 510, ranked by callers