hub / github.com/abeltavares/batch-data-pipeline

github.com/abeltavares/batch-data-pipeline @main

Chat with this repo

repository ↗ · DeepWiki ↗ · + Follow

72 symbols 329 edges 18 files 65 documented · 90% ● updated 9mo ago★ 901 open issues

Browse by type

⨍ Functions 59 ◇ Types & classes 12 ↳ Endpoints 1

What it actually does AI analysis from the code graph — generated when you open this

loading…

Core symbols most depended-on inside this repo

↓ 8 filesget_config

Get configuration value.

src/config.py

↓ 6 filesget_logger

Get configured logger.

src/utils/logger.py

↓ 5 filesStorageClient

S3/MinIO storage client.

src/utils/storage.py

↓ 4 filesBaseProcessor

Abstract base class for all processors. Enforces interface contract and handles common operations. Design Pattern: Interface …

src/processors/base_processor.py

↓ 3 filesDuckDBClient

DuckDB client with S3 configuration.

src/utils/database.py

↓ 3 filesConfig

Pipeline configuration singleton.

src/config.py

browse all functions →

Shape

Method 45

Function 14

Class 12

Route 1

Languages

Python100%

Modules by API surface

src/processors/gold_processor.py9 symbols

airflow/dags/sales_pipeline.py9 symbols

src/processors/base_processor.py7 symbols

src/models/schemas.py7 symbols

src/utils/storage.py6 symbols

src/config.py6 symbols

src/utils/database.py5 symbols

src/processors/silver_processor.py5 symbols

src/generators/sales_generator.py5 symbols

src/utils/metrics.py4 symbols

src/processors/quality_checker.py4 symbols

src/processors/bronze_processor.py4 symbols

Datastores touched

airflowDatabase · 1 repos

For agents

$ claude mcp add batch-data-pipeline \
  -- python -m otcore.mcp_server <graph>

⬇ download graph artifact

github.com/abeltavares/batch-data-pipeline @main

Core symbols most depended-on inside this repo

Shape

Languages

Modules by API surface

Datastores touched

For agents

Ask about this repo answers extend the page