MCPcopy Index your code
hub / github.com/dadoonet/fscrawler

github.com/dadoonet/fscrawler @fscrawler-2.9

Chat with this repo
repository ↗ · DeepWiki ↗ · release fscrawler-2.9 ↗ · + Follow
1,513 symbols 6,166 edges 179 files 272 documented · 18%
What it actually does AI analysis from the code graph — generated when you open this
loading…
README

File System Crawler for Elasticsearch

Welcome to the FS Crawler for Elasticsearch

This crawler helps to index binary documents such as PDF, Open Office, MS Office.

Main features:

  • Local file system (or a mounted drive) crawling and index new files, update existing ones and removes old ones.
  • Remote file system over SSH/FTP crawling.
  • REST interface to let you "upload" your binary documents to elasticsearch.

You need to install a version matching your Elasticsearch version:

Elasticsearch FS Crawler Released Docs
6.x, 7.x 2.9-SNAPSHOT 2.9-SNAPSHOT
6.x, 7.x 2.8 2021-12-13 2.8
6.x, 7.x 2.7 2021-08-05 2.7
2.x, 5.x, 6.x 2.6 2019-01-09 2.6
2.x, 5.x, 6.x 2.5 2018-08-04 2.5
2.x, 5.x, 6.x 2.4 2017-08-11 2.4
2.x, 5.x, 6.x 2.3 2017-07-10 2.3
1.x, 2.x, 5.x 2.2 2017-02-03 2.2
1.x, 2.x, 5.x 2.1 2016-07-26 2.1
es-2.0 2.0.0 2015-10-30 2.0.0

Build and Quality Status

Maven Central Build Travis Documentation Status Code Quality: Java Total Alerts

Lines Duplicated Lines Maintainability Technical Debt Reliability

Vulnerabilities Bugs Quality Gate Code Smells Coverage

The guide has been moved to ReadTheDocs.

License

Read more about the License.

Thanks

Thanks to JetBrains for the IntelliJ IDEA License!

Extension points exported contracts — how you extend this code

FsCrawlerService (Interface)
(no doc) [6 implementers]
core/src/main/java/fr/pilato/elasticsearch/crawler/fs/service/FsCrawlerService.java
ElasticsearchClient (Interface)
Simple Elasticsearch client over HTTP or HTTPS. Only needed methods are exposed. [4 implementers]
elasticsearch-client/elasticsearch-client-base/src/main/java/fr/pilato/elasticsearch/crawler/fs/client/ElasticsearchClient.java
Engine (Interface)
Defines an Engine which is responsible to execute a bulk operation
framework/src/main/java/fr/pilato/elasticsearch/crawler/fs/framework/bulk/Engine.java
FsCrawlerDocumentService (Interface)
(no doc) [4 implementers]
core/src/main/java/fr/pilato/elasticsearch/crawler/fs/service/FsCrawlerDocumentService.java
WorkplaceSearchClient (Interface)
(no doc) [7 implementers]
elasticsearch-client/elasticsearch-client-base/src/main/java/fr/pilato/elasticsearch/crawler/fs/client/WorkplaceSearchClient.java
Listener (Interface)
(no doc) [1 implementers]
framework/src/main/java/fr/pilato/elasticsearch/crawler/fs/framework/bulk/FsCrawlerBulkProcessor.java
FsCrawlerManagementService (Interface)
(no doc) [2 implementers]
core/src/main/java/fr/pilato/elasticsearch/crawler/fs/service/FsCrawlerManagementService.java
FsCrawlerOperation (Interface)
(no doc)
framework/src/main/java/fr/pilato/elasticsearch/crawler/fs/framework/bulk/FsCrawlerOperation.java

Core symbols most depended-on inside this repo

build
called by 165
beans/src/main/java/fr/pilato/elasticsearch/crawler/fs/beans/FsJob.java
read
called by 138
beans/src/main/java/fr/pilato/elasticsearch/crawler/fs/beans/FsJobFileHandler.java
get
called by 118
core/src/main/java/fr/pilato/elasticsearch/crawler/fs/service/FsCrawlerDocumentService.java
getFs
called by 114
settings/src/main/java/fr/pilato/elasticsearch/crawler/fs/settings/FsSettings.java
withIndex
called by 104
elasticsearch-client/elasticsearch-client-base/src/main/java/fr/pilato/elasticsearch/crawler/fs/client/ESSearchRequest.java
getMeta
called by 103
beans/src/main/java/fr/pilato/elasticsearch/crawler/fs/beans/Doc.java
getElasticsearch
called by 78
settings/src/main/java/fr/pilato/elasticsearch/crawler/fs/settings/FsSettings.java
isIndexable
called by 72
framework/src/main/java/fr/pilato/elasticsearch/crawler/fs/framework/FsCrawlerUtil.java

Shape

Method 1,313
Class 189
Interface 9
Enum 1
Function 1

Languages

Java100%
Python1%

Modules by API surface

settings/src/main/java/fr/pilato/elasticsearch/crawler/fs/settings/Fs.java81 symbols
beans/src/main/java/fr/pilato/elasticsearch/crawler/fs/beans/Meta.java54 symbols
settings/src/main/java/fr/pilato/elasticsearch/crawler/fs/settings/Elasticsearch.java38 symbols
elasticsearch-client/elasticsearch-client-v6/src/main/java/fr/pilato/elasticsearch/crawler/fs/client/v6/ElasticsearchClientV6.java38 symbols
framework/src/main/java/fr/pilato/elasticsearch/crawler/fs/framework/FsCrawlerUtil.java37 symbols
elasticsearch-client/elasticsearch-client-v7/src/main/java/fr/pilato/elasticsearch/crawler/fs/client/v7/ElasticsearchClientV7.java37 symbols
settings/src/main/java/fr/pilato/elasticsearch/crawler/fs/settings/WorkplaceSearch.java32 symbols
settings/src/main/java/fr/pilato/elasticsearch/crawler/fs/settings/Server.java27 symbols
tika/src/test/java/fr/pilato/elasticsearch/crawler/fs/tika/TikaDocParserTest.java26 symbols
settings/src/main/java/fr/pilato/elasticsearch/crawler/fs/settings/Ocr.java26 symbols
settings/src/main/java/fr/pilato/elasticsearch/crawler/fs/settings/FsSettings.java26 symbols
framework/src/main/java/fr/pilato/elasticsearch/crawler/fs/framework/TimeValue.java26 symbols

For agents

$ claude mcp add fscrawler \
  -- python -m otcore.mcp_server <graph>

⬇ download graph artifact