MCPcopy Create free account

hub / github.com/CrawlScript/WebCollector / functions

Functions510 in github.com/CrawlScript/WebCollector

↓ 52 callersMethodmeta
(JsonObject metaData)
src/main/java/cn/edu/hfut/dmic/webcollector/model/MetaSetter.java:6
↓ 50 callersMethodadd
(String url)
src/main/java/cn/edu/hfut/dmic/webcollector/model/Links.java:54
↓ 33 callersMethodget
(int index)
src/main/java/cn/edu/hfut/dmic/webcollector/model/Links.java:167
↓ 28 callersMethodgetConf
()
src/main/java/cn/edu/hfut/dmic/webcollector/conf/Configured.java:24
↓ 23 callersMethodset
(String key, Object value)
src/main/java/cn/edu/hfut/dmic/webcollector/conf/Configuration.java:53
↓ 22 callersMethodsize
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/Links.java:171
↓ 22 callersMethodstart
开始爬取,迭代次数为depth @param depth 迭代次数 @throws Exception 异常
src/main/java/cn/edu/hfut/dmic/webcollector/crawler/Crawler.java:91
↓ 22 callersMethodurl
返回网页的url @return 网页的url
src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:353
↓ 20 callersMethodtype
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatum.java:128
↓ 18 callersMethodaddSeedAndReturn
(CrawlDatum datum, boolean force)
src/main/java/cn/edu/hfut/dmic/webcollector/crawler/Crawler.java:288
↓ 18 callersMethodget
(String key)
src/main/java/cn/edu/hfut/dmic/webcollector/conf/Configuration.java:73
↓ 17 callersMethodaddSeed
添加种子任务 @param datum 种子任务 @param force 如果添加的种子是已爬取的任务,当force为true时,会强制注入种子,当force为false时,会忽略该种子
src/main/java/cn/edu/hfut/dmic/webcollector/crawler/Crawler.java:169
↓ 17 callersMethodkey
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:505
↓ 17 callersMethodput
(RocksDB rocksDB, String key,String value)
src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksDBUtils.java:103
↓ 16 callersMethodselect
(URI uri)
src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoRandomProxyCrawler.java:45
↓ 15 callersMethodparse
(String json)
src/main/java/cn/edu/hfut/dmic/webcollector/util/GsonUtils.java:10
↓ 13 callersMethodaddAndReturn
(String url)
src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatums.java:90
↓ 12 callersMethodadd
(CrawlDatum datum)
src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatums.java:56
↓ 12 callersMethodencode
(String charset, String text)
src/test/java/cn/edu/hfut/dmic/webcollector/util/CharsetDetectorTest.java:51
↓ 12 callersMethodtoString
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/Links.java:196
↓ 11 callersMethodclose
()
src/main/java/cn/edu/hfut/dmic/webcollector/crawldb/DBManager.java:47
↓ 11 callersMethodmeta
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatum.java:189
↓ 11 callersMethodselect
(String cssSelector)
src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:274
↓ 10 callersMethodaddRegex
添加URL正则约束 @param urlRegex URL正则约束
src/main/java/cn/edu/hfut/dmic/webcollector/crawler/AutoParseCrawler.java:100
↓ 10 callersMethodclose
()
src/main/java/cn/edu/hfut/dmic/webcollector/plugin/berkeley/BerkeleyDBManager.java:125
↓ 10 callersMethoddoc
返回网页解析后的DOM树(Jsoup的Document对象) 已废弃,使用doc()方法代替 @return 网页解析后的DOM树
src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:403
↓ 10 callersMethodhtml
返回网页的源码字符串 @return 网页的源码字符串
src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:363
↓ 10 callersMethodmetaAsInt
(String key)
src/main/java/cn/edu/hfut/dmic/webcollector/model/MetaGetter.java:8
↓ 10 callersMethodsetThreads
设置爬虫的线程数 @param threads 爬虫的线程数
src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/Fetcher.java:413
↓ 9 callersMethodattr
获取网页中满足指定css选择器的所有元素的指定属性的集合 例如通过attr("img[src]","abs:src")可获取网页中第一个图片的链接 @param cssSelector @param attrName @return
src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:157
↓ 9 callersMethodclose
()
src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksDBManager.java:110
↓ 9 callersMethodcode
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:510
↓ 9 callersMethodget
(RocksDB rocksDB, String key)
src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksDBUtils.java:107
↓ 8 callersMethodlinks
(boolean parseImg)
src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:162
↓ 8 callersMethodmatchType
判断当前Page(CrawlDatum)的type是否为type @param type @return 是否相等
src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:88
↓ 8 callersMethodset
(int value)
src/main/java/cn/edu/hfut/dmic/webcollector/util/Counter.java:44
↓ 7 callersMethodcheckMethod
(Method method)
src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/VisitorMethodDispatcher.java:36
↓ 7 callersMethodfail
(String message)
src/main/java/cn/edu/hfut/dmic/webcollector/util/ExceptionUtils.java:24
↓ 7 callersMethodisEmpty
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/Links.java:187
↓ 7 callersMethodselectText
(String cssSelector, int index)
src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:215
↓ 7 callersMethodurl
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatum.java:154
↓ 6 callersMethodasJsonArray
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatum.java:299
↓ 6 callersMethodgetResponse
(String url)
src/main/java/cn/edu/hfut/dmic/webcollector/net/Requester.java:30
↓ 6 callersMethodguessEncoding
根据字节数组,猜测可能的字符集,如果检测失败,返回utf-8 @param content 待检测的字节数组 @return 可能的字符集,如果检测失败,返回utf-8
src/main/java/cn/edu/hfut/dmic/webcollector/util/CharsetDetector.java:110
↓ 6 callersMethodguessEncodingByMozilla
根据字节数组,猜测可能的字符集,如果检测失败,返回utf-8 @param bytes 待检测的字节数组 @return 可能的字符集,如果检测失败,返回utf-8
src/main/java/cn/edu/hfut/dmic/webcollector/util/CharsetDetector.java:92
↓ 6 callersMethodlocation
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:514
↓ 6 callersMethodsetTo
(Configured from, Object... targets)
src/main/java/cn/edu/hfut/dmic/webcollector/util/ConfigurationUtils.java:14
↓ 6 callersMethodstrToEntry
(String str)
src/main/java/cn/edu/hfut/dmic/webcollector/plugin/berkeley/BerkeleyDBUtils.java:56
↓ 5 callersMethodaddRule
添加一个正则规则 正则规则有两种,正正则和反正则 URL符合正则规则需要满足下面条件: 1.至少能匹配一条正正则 2.不能和任何反正则匹配 正正则示例:+a. c是一条正正则,正则的内容为a. c,起始加号表示正正则 反正则示例:-a. c时一条反正则,正则的内容为a. c,起始减号表示反正则 如果
src/main/java/cn/edu/hfut/dmic/webcollector/util/RegexRule.java:67
↓ 5 callersMethodcontent
返回网页/文件的内容 @return 网页/文件的内容
src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:339
↓ 5 callersMethodcopyMeta
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/MetaGetter.java:12
↓ 5 callersMethodkey
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatum.java:239
↓ 5 callersMethodnext
()
src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksDBReader.java:51
↓ 5 callersMethodopenCrawldbDatabase
(String crawlPath)
src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksDBUtils.java:59
↓ 5 callersMethodput
(Database database,String key,String value)
src/main/java/cn/edu/hfut/dmic/webcollector/plugin/berkeley/BerkeleyDBUtils.java:52
↓ 5 callersMethodsetTopN
(Integer topN)
src/main/java/cn/edu/hfut/dmic/webcollector/conf/Configuration.java:105
↓ 4 callersMethodcontentType
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:394
↓ 4 callersMethodgetContentElementByDoc
(Document doc)
src/main/java/cn/edu/hfut/dmic/contentextractor/ContentExtractor.java:436
↓ 4 callersMethodgetSize
()
src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/Fetcher.java:100
↓ 4 callersMethodgetStatus
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatum.java:180
↓ 4 callersMethodmatchUrl
判断当前Page的URL是否和输入正则匹配 @param urlRegex @return
src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:69
↓ 4 callersMethodopen
(String dbPath)
src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksDBUtils.java:87
↓ 4 callersMethodregex
(String regex, int group, String defaultResult)
src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:287
↓ 4 callersMethodremove
(int index)
src/main/java/cn/edu/hfut/dmic/webcollector/model/Links.java:175
↓ 4 callersMethodsatisfy
判断输入字符串是否符合正则规则 @param str 输入的字符串 @return 输入字符串是否符合正则规则
src/main/java/cn/edu/hfut/dmic/webcollector/util/RegexRule.java:112
↓ 4 callersMethodselectTextList
(String cssSelector)
src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:206
↓ 4 callersMethodsetExecuteInterval
(Integer executeInterval)
src/main/java/cn/edu/hfut/dmic/webcollector/conf/Configuration.java:133
↓ 4 callersMethodwrite
(String fileName, String contentStr, String charset)
src/main/java/cn/edu/hfut/dmic/webcollector/util/FileUtils.java:60
↓ 3 callersMethodaddByRegex
(Element ele, RegexRule regexRule, boolean parseSrc)
src/main/java/cn/edu/hfut/dmic/webcollector/model/Links.java:134
↓ 3 callersMethodcreateCrawlDatum
(byte[] key, byte[] value)
src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksDBUtils.java:124
↓ 3 callersMethodcreateCrawlDatum
(DatabaseEntry key,DatabaseEntry value)
src/main/java/cn/edu/hfut/dmic/webcollector/plugin/berkeley/BerkeleyDBUtils.java:60
↓ 3 callersMethodcreateDefaultDBOptions
()
src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksDBUtils.java:83
↓ 3 callersMethoddatumToString
(CrawlDatum datum)
src/main/java/cn/edu/hfut/dmic/webcollector/util/CrawlDatumFormater.java:39
↓ 3 callersMethodgetContentElement
()
src/main/java/cn/edu/hfut/dmic/contentextractor/ContentExtractor.java:149
↓ 3 callersMethodgetExecuteCount
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatum.java:172
↓ 3 callersMethodinject
(CrawlDatum datum)
src/main/java/cn/edu/hfut/dmic/webcollector/crawldb/Injector.java:27
↓ 3 callersMethodnext
return null if there is no CrawlDatum to generate @return
src/main/java/cn/edu/hfut/dmic/webcollector/crawldb/Generator.java:49
↓ 3 callersMethodsetRequester
(Requester requester)
src/main/java/cn/edu/hfut/dmic/webcollector/crawler/AutoParseCrawler.java:161
↓ 3 callersMethodsetStatus
(int status)
src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatum.java:184
↓ 3 callersMethodsize
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatums.java:180
↓ 3 callersMethodstrToKeyOrValue
(String str)
src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksDBUtils.java:120
↓ 2 callersMethodaddPositive
添加一个正正则规则 @param positiveregex @return 自身
src/main/java/cn/edu/hfut/dmic/webcollector/util/RegexRule.java:90
↓ 2 callersMethodbriefInfo
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatum.java:225
↓ 2 callersMethodcharset
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:493
↓ 2 callersMethodclear
()
src/main/java/cn/edu/hfut/dmic/webcollector/plugin/ram/RamDBManager.java:49
↓ 2 callersMethodcreateGenerator
()
src/main/java/cn/edu/hfut/dmic/webcollector/crawldb/DBManager.java:36
↓ 2 callersMethoddeleteDir
(File dir)
src/main/java/cn/edu/hfut/dmic/webcollector/util/FileUtils.java:33
↓ 2 callersMethodexecute
(CrawlDatum datum,CrawlDatums next)
src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/Executor.java:27
↓ 2 callersMethodfromJsonArray
(String crawlDatumKey, JsonArray jsonArray)
src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatum.java:315
↓ 2 callersMethodget
(int index)
src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatums.java:176
↓ 2 callersMethodget
()
src/main/java/cn/edu/hfut/dmic/webcollector/util/Counter.java:48
↓ 2 callersMethodgetContent
()
src/main/java/cn/edu/hfut/dmic/contentextractor/News.java:51
↓ 2 callersMethodgetData
()
src/main/java/cn/edu/hfut/dmic/webcollector/conf/Configuration.java:86
↓ 2 callersMethodgetDefault
()
src/main/java/cn/edu/hfut/dmic/webcollector/conf/Configuration.java:194
↓ 2 callersMethodgetExecuteTime
()
src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatum.java:164
↓ 2 callersMethodgetFetchPath
(String crawlPath)
src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksDBUtils.java:44
↓ 2 callersMethodgetFullMethodName
(Method method)
src/main/java/cn/edu/hfut/dmic/webcollector/util/ReflectionUtils.java:41
↓ 2 callersMethodgetInteger
(String key)
src/main/java/cn/edu/hfut/dmic/webcollector/conf/Configuration.java:62
↓ 2 callersMethodgetLinkPath
(String crawlPath)
src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksDBUtils.java:49
↓ 2 callersMethodgetNewsByDoc
(Document doc)
src/main/java/cn/edu/hfut/dmic/contentextractor/ContentExtractor.java:489
next →1–100 of 510, ranked by callers