hub / github.com/CrawlScript/WebCollector

github.com/CrawlScript/WebCollector @main

Chat with this repo

repository ↗ · DeepWiki ↗ · + Follow

607 symbols 1,594 edges 94 files 133 documented · 22% ● updated 6mo ago★ 3,08348 open issues

Browse by type

⨍ Functions 510◇ Types & classes 97

What it actually does AI analysis from the code graph — generated when you open this

loading…

Extension points exported contracts — how you extend this code

Visitor (Interface)

@author hu [20 implementers]

src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/Visitor.java

Injector (Interface)

@author hu [6 implementers]

src/main/java/cn/edu/hfut/dmic/webcollector/crawldb/Injector.java

NextFilter (Interface)

@author hu [4 implementers]

src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/NextFilter.java

Requester (Interface)

@author hu [4 implementers]

src/main/java/cn/edu/hfut/dmic/webcollector/net/Requester.java

SegmentWriter (Interface)

爬取过程中，写入爬取历史、网页Content、解析信息的Writer @author hu [4 implementers]

src/main/java/cn/edu/hfut/dmic/webcollector/crawldb/SegmentWriter.java

browse all types & interfaces →

Core symbols most depended-on inside this repo

↓ 23 filesCrawler

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/crawler/Crawler.java

↓ 20 filesVisitor

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/Visitor.java

↓ 20 filesPage

Page是爬取过程中，内存中保存网页爬取信息的一个容器，Page只在内存中存放，用于保存一些网页信息，方便用户进行自定义网页解析之类的操作。 @author hu

src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java

↓ 14 filesConfigured

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/conf/Configured.java

↓ 13 filesConfiguration

@author hu

src/main/java/cn/edu/hfut/dmic/webcollector/conf/Configuration.java

↓ 12 filesBreadthCrawler

cn.edu.hfut.dmic.webcollector.plugin.rocks.BreadthCrawler是基于RocksDB的插件,于2.72版重新设计 …

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/BreadthCrawler.java

browse all functions →

Shape

Method 510

Class 87

Interface 10

Languages

Java100%

Modules by API surface

src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java44 symbols

src/main/java/cn/edu/hfut/dmic/webcollector/conf/Configuration.java39 symbols

src/main/java/cn/edu/hfut/dmic/contentextractor/ContentExtractor.java28 symbols

src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatum.java27 symbols

src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/Fetcher.java26 symbols

src/main/java/cn/edu/hfut/dmic/webcollector/crawler/Crawler.java24 symbols

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksDBUtils.java17 symbols

src/main/java/cn/edu/hfut/dmic/webcollector/model/Links.java16 symbols

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/rocks/RocksDBManager.java14 symbols

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/berkeley/BerkeleyDBManager.java14 symbols

src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatums.java14 symbols

src/main/java/cn/edu/hfut/dmic/webcollector/plugin/ram/RamDBManager.java13 symbols

For agents

$ claude mcp add WebCollector \
  -- python -m otcore.mcp_server <graph>

⬇ download graph artifact

github.com/CrawlScript/WebCollector @main

Extension points exported contracts — how you extend this code

Core symbols most depended-on inside this repo

Shape

Languages

Modules by API surface

For agents

Ask about this repo answers extend the page