Functions — github.com/CrawlScript/WebCollector

hub / github.com/CrawlScript/WebCollector / functions

Functions510 in github.com/CrawlScript/WebCollector

⨍ Functions 510◇ Types & classes 97

↓ 2 callersMethodgetString

(String key)

src/main/java/cn/edu/hfut/dmic/webcollector/conf/Configuration.java:58

↓ 2 callersMethoditerator

()

src/main/java/cn/edu/hfut/dmic/webcollector/model/Links.java:71

↓ 2 callersMethodmetaAsLong

(String key)

src/main/java/cn/edu/hfut/dmic/webcollector/model/MetaGetter.java:11

↓ 2 callersMethodopen

()

src/main/java/cn/edu/hfut/dmic/webcollector/crawldb/DBManager.java:45

↓ 2 callersMethodopenFetchDatabase

(String crawlPath)

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksDBUtils.java:63

↓ 2 callersMethodopenLinkDatabase

(String crawlPath)

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksDBUtils.java:67

↓ 2 callersMethodremoveSuccessCode

(int successCode)

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/net/OkHttpRequester.java:43

↓ 2 callersMethodsetConf

(Configuration conf)

src/main/java/cn/edu/hfut/dmic/webcollector/conf/Configured.java:25

↓ 2 callersMethodsetExecuteTime

(long executeTime)

src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatum.java:168

↓ 2 callersMethodsetNextFilter

(NextFilter nextFilter)

src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/Fetcher.java:430

↓ 2 callersMethodstrSim

(String a, String b)

src/main/java/cn/edu/hfut/dmic/contentextractor/ContentExtractor.java:262

↓ 2 callersMethodtestInject

(DBManager dbManager)

src/test/java/cn/edu/hfut/dmic/webcollector/util/DBManagerTest.java:18

↓ 2 callersMethodtoString

()

src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatum.java:256

↓ 2 callersMethodtoString

()

src/main/java/cn/edu/hfut/dmic/contentextractor/News.java:74

↓ 2 callersMethodwriteDatum

(RocksDB rocksDB,CrawlDatum datum)

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksDBUtils.java:97

↓ 2 callersMethodwriteDatum

(Database database,CrawlDatum datum)

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/berkeley/BerkeleyDBUtils.java:46

↓ 1 callersMethodaddBySelector

添加ele中，满足选择器的元素中的链接选择器cssSelector必须定位到具体的超链接例如我们想抽取id为content的div中的所有超链接，这里就要将cssSelector定义为div[id=content] a

src/main/java/cn/edu/hfut/dmic/webcollector/model/Links.java:114

↓ 1 callersMethodaddFetchItem

(FetchItem item)

src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/Fetcher.java:104

↓ 1 callersMethodaddFromElement

(Element ele)

src/main/java/cn/edu/hfut/dmic/webcollector/model/Links.java:93

↓ 1 callersMethodaddNegative

添加一个反正则规则 @param negativeregex @return 自身

src/main/java/cn/edu/hfut/dmic/webcollector/util/RegexRule.java:101

↓ 1 callersMethodaddSocksProxy

(String ip, int port)

src/main/java/cn/edu/hfut/dmic/webcollector/net/Proxies.java:70

↓ 1 callersMethodafterStop

()

src/main/java/cn/edu/hfut/dmic/webcollector/crawler/Crawler.java:151

↓ 1 callersMethodbuildMessage

(Visitor visitor, Method method)

src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/Visitor.java:88

↓ 1 callersMethodclean

()

src/main/java/cn/edu/hfut/dmic/contentextractor/ContentExtractor.java:69

↓ 1 callersMethodclear

()

src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/Fetcher.java:96

↓ 1 callersMethodclear

()

src/main/java/cn/edu/hfut/dmic/webcollector/model/Links.java:183

↓ 1 callersMethodclear

()

src/main/java/cn/edu/hfut/dmic/webcollector/crawldb/DBManager.java:34

↓ 1 callersMethodclose

()

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/ram/RamGenerator.java:51

↓ 1 callersMethodclose

()

src/main/java/cn/edu/hfut/dmic/webcollector/crawldb/Generator.java:104

↓ 1 callersMethodcloseGenerator

()

src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/Fetcher.java:156

↓ 1 callersMethodcloseSegmentWriter

()

src/main/java/cn/edu/hfut/dmic/webcollector/crawldb/SegmentWriter.java:36

↓ 1 callersMethodcode

()

src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatum.java:136

↓ 1 callersMethodcomputeInfo

(Node node)

src/main/java/cn/edu/hfut/dmic/contentextractor/ContentExtractor.java:73

↓ 1 callersMethodcomputeScore

(Element tag)

src/main/java/cn/edu/hfut/dmic/contentextractor/ContentExtractor.java:122

↓ 1 callersMethodcomputeVar

(ArrayList<Integer> data)

src/main/java/cn/edu/hfut/dmic/contentextractor/ContentExtractor.java:129

↓ 1 callersMethodcopy

()

src/main/java/cn/edu/hfut/dmic/webcollector/conf/Configuration.java:95

↓ 1 callersMethodcrawlDatum

()

src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:437

↓ 1 callersMethodcreateBingUrl

construct the Bing Search url by the search keyword and the pageIndex @param keyword @param pageIndex @return the constructed url @throws Exception

src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoRedirectCrawler.java:79

↓ 1 callersMethodcreateBingUrl

construct the Bing Search url by the search keyword and the pageIndex @param keyword @param pageIndex @return the constructed url @throws Exception

src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoAnnotatedRedirectCrawler.java:78

↓ 1 callersMethodcreateBingUrl

construct the Bing Search url by the search keyword and the pageIndex @param keyword @param pageIndex @return the constructed url @throws Exception

src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoBingCrawler.java:152

↓ 1 callersMethodcreateBingUrl

construct the Bing Search url by the search keyword and the pageIndex @param keyword @param pageIndex @return the constructed url @throws Exception

src/main/java/cn/edu/hfut/dmic/webcollector/example/DemoAnnotatedBingCrawler.java:157

↓ 1 callersMethodcreateDataSource

(String url, String username, String password, int initialSize, int maxPoolSize)

src/main/java/cn/edu/hfut/dmic/webcollector/util/MysqlHelper.java:41

↓ 1 callersMethodcreateDefaultDBConfig

()

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/berkeley/BerkeleyDBUtils.java:39

↓ 1 callersMethodcreateOkHttpClientBuilder

()

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/net/OkHttpRequester.java:58

↓ 1 callersMethodcreateRequestBuilder

(CrawlDatum crawlDatum)

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/net/OkHttpRequester.java:68

↓ 1 callersMethodcreateSuccessCodeSet

()

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/net/OkHttpRequester.java:48

↓ 1 callersMethoddestroyFetchDatabase

(String crawlPath)

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksDBUtils.java:72

↓ 1 callersMethoddestroyLinkDatabase

(String crawlPath)

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksDBUtils.java:76

↓ 1 callersMethoddispatch

(Page page, CrawlDatums next)

src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/VisitorMethodDispatcher.java:199

↓ 1 callersMethoddump

()

src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/Fetcher.java:119

↓ 1 callersMethodfetchAll

抓取当前所有任务，会阻塞到爬取完成 @throws IOException 异常

src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/Fetcher.java:304

↓ 1 callersMethodfilter

if the crawler visit http://a.com/ and detect http://a.com/b.html then nextItem = http://a.com/b.html and referer = http://a.com/ if you want to filte

src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/NextFilter.java:38

↓ 1 callersMethodfilter

return datum if you want to generate datum return null if you want to filter datum @param datum @return

src/main/java/cn/edu/hfut/dmic/webcollector/crawldb/GeneratorFilter.java:13

↓ 1 callersMethodgenerate

()

src/main/java/cn/edu/hfut/dmic/webcollector/util/FileIdGenerator.java:42

↓ 1 callersMethodgetAutoDetectImg

()

src/main/java/cn/edu/hfut/dmic/webcollector/conf/Configuration.java:182

↓ 1 callersMethodgetByIndex

(List<T> list, int index)

src/main/java/cn/edu/hfut/dmic/webcollector/util/ListUtils.java:7

↓ 1 callersMethodgetConnectTimeout

()

src/main/java/cn/edu/hfut/dmic/webcollector/conf/Configuration.java:119

↓ 1 callersMethodgetContentByHtml

(String html)

src/main/java/cn/edu/hfut/dmic/contentextractor/ContentExtractor.java:469

↓ 1 callersMethodgetContentElementByHtml

(String html)

src/main/java/cn/edu/hfut/dmic/contentextractor/ContentExtractor.java:442

↓ 1 callersMethodgetCrawldbPath

(String crawlPath)

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksDBUtils.java:54

↓ 1 callersMethodgetDate

(Element contentElement)

src/main/java/cn/edu/hfut/dmic/contentextractor/ContentExtractor.java:234

↓ 1 callersMethodgetDefaultCookie

()

src/main/java/cn/edu/hfut/dmic/webcollector/conf/Configuration.java:173

↓ 1 callersMethodgetDefaultUserAgent

()

src/main/java/cn/edu/hfut/dmic/webcollector/conf/Configuration.java:166

↓ 1 callersMethodgetExecuteInterval

()

src/main/java/cn/edu/hfut/dmic/webcollector/conf/Configuration.java:130

↓ 1 callersMethodgetFetchItem

()

src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/Fetcher.java:112

↓ 1 callersMethodgetMethodAccessibility

(Method method)

src/main/java/cn/edu/hfut/dmic/webcollector/util/ReflectionUtils.java:27

↓ 1 callersMethodgetMethodByCode

(Page page)

src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/VisitorMethodDispatcher.java:170

↓ 1 callersMethodgetMethodByType

(Page page)

src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/VisitorMethodDispatcher.java:175

↓ 1 callersMethodgetMethodByUrlRegex

(Page page)

src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/VisitorMethodDispatcher.java:180

↓ 1 callersMethodgetMethodByUrlRegexRule

(Page page)

src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/VisitorMethodDispatcher.java:189

↓ 1 callersMethodgetMethodDeclaration

(Method method)

src/main/java/cn/edu/hfut/dmic/webcollector/util/ReflectionUtils.java:7

↓ 1 callersMethodgetNews

()

src/main/java/cn/edu/hfut/dmic/contentextractor/ContentExtractor.java:171

↓ 1 callersMethodgetNewsByHtml

(String html)

src/main/java/cn/edu/hfut/dmic/contentextractor/ContentExtractor.java:495

↓ 1 callersMethodgetNewsByUrl

(String url)

src/main/java/cn/edu/hfut/dmic/contentextractor/ContentExtractor.java:507

↓ 1 callersMethodgetOrDefault

(String key, T defaultValue)

src/main/java/cn/edu/hfut/dmic/webcollector/conf/Configuration.java:77

↓ 1 callersMethodgetReadTimeout

()

src/main/java/cn/edu/hfut/dmic/webcollector/conf/Configuration.java:126

↓ 1 callersMethodgetThreadKiller

()

src/main/java/cn/edu/hfut/dmic/webcollector/conf/Configuration.java:137

↓ 1 callersMethodgetTime

()

src/main/java/cn/edu/hfut/dmic/contentextractor/News.java:66

↓ 1 callersMethodgetTime

(Element contentElement)

src/main/java/cn/edu/hfut/dmic/contentextractor/ContentExtractor.java:200

↓ 1 callersMethodgetTitle

()

src/main/java/cn/edu/hfut/dmic/contentextractor/News.java:43

↓ 1 callersMethodgetTitle

(final Element contentElement)

src/main/java/cn/edu/hfut/dmic/contentextractor/ContentExtractor.java:280

↓ 1 callersMethodgetTitleByEditDistance

(Element contentElement)

src/main/java/cn/edu/hfut/dmic/contentextractor/ContentExtractor.java:341

↓ 1 callersMethodgetTopN

()

src/main/java/cn/edu/hfut/dmic/webcollector/conf/Configuration.java:102

↓ 1 callersMethodgetTotalGenerate

()

src/main/java/cn/edu/hfut/dmic/webcollector/crawldb/Generator.java:100

↓ 1 callersMethodgetUrl

()

src/main/java/cn/edu/hfut/dmic/contentextractor/News.java:35

↓ 1 callersMethodgetWaitThreadEndTime

()

src/main/java/cn/edu/hfut/dmic/webcollector/conf/Configuration.java:145

↓ 1 callersMethodguessEncodingByNutch

(byte[] content)

src/main/java/cn/edu/hfut/dmic/webcollector/util/CharsetDetector.java:45

↓ 1 callersMethodincrExecuteCount

(int count)

src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatum.java:121

↓ 1 callersMethodinitLastId

()

src/main/java/cn/edu/hfut/dmic/webcollector/util/FileIdGenerator.java:30

↓ 1 callersMethodinitSegmentWriter

()

src/main/java/cn/edu/hfut/dmic/webcollector/crawldb/SegmentWriter.java:30

↓ 1 callersMethodinject

()

src/main/java/cn/edu/hfut/dmic/webcollector/crawler/Crawler.java:72

↓ 1 callersMethodinjectForcedSeeds

()

src/main/java/cn/edu/hfut/dmic/webcollector/crawler/Crawler.java:76

↓ 1 callersMethodisDBExists

()

src/main/java/cn/edu/hfut/dmic/webcollector/crawldb/DBManager.java:32

↓ 1 callersMethodisEmpty

()

src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatums.java:196

↓ 1 callersMethodjsonObject

()

src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:105

↓ 1 callersMethodkeyOrValueToStr

(byte[] keyOrValue)

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksDBUtils.java:116

↓ 1 callersMethodlcs

(String x, String y)

src/main/java/cn/edu/hfut/dmic/contentextractor/ContentExtractor.java:376

↓ 1 callersMethodlocation

()

src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatum.java:145

↓ 1 callersMethodmatchUrlRegexRule

判断当前Page的URL是否和输入正则规则匹配 @param urlRegexRule @return

src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java:78

↓ 1 callersMethodmd5

(byte[] original)

src/main/java/cn/edu/hfut/dmic/webcollector/util/MD5Utils.java:32

← previous next →101–200 of 510, ranked by callers