Types & classes — github.com/abeltavares/batch-data-pipeline

CodeHub MCPcopy Create free account

hub / github.com/abeltavares/batch-data-pipeline / types & classes

Types & classes12 in github.com/abeltavares/batch-data-pipeline

⨍ Functions 59 ◇ Types & classes 12 ↳ Endpoints 1

↓ 5 callersClassDuckDBClient

DuckDB client with S3 configuration.

src/utils/database.py:9

↓ 2 callersClassBronzeProcessor

Bronze layer processor - ingests raw data without validation. Accepts dirty data intentionally for downstream cleaning.

src/processors/bronze_processor.py:9

↓ 1 callersClassConfig

Pipeline configuration singleton.

src/config.py:8

↓ 1 callersClassCustomer

Customer dimension schema.

src/models/schemas.py:54

↓ 1 callersClassGoldProcessor

Gold layer processor - creates business aggregations. Uses ThreadPoolExecutor for parallel table processing.

src/processors/gold_processor.py:15

↓ 1 callersClassProduct

Product dimension schema.

src/models/schemas.py:45

↓ 1 callersClassQualityChecker

Quality checker processor - validates Silver data using Soda Core. Implements the same interface as other processors.

src/processors/quality_checker.py:11

↓ 1 callersClassSalesDataGenerator

Generate synthetic sales data with realistic data quality issues.

src/generators/sales_generator.py:13

↓ 1 callersClassSilverProcessor

Silver layer processor - cleans, validates, and deduplicates data. Writes to Delta Lake for ACID transactions.

src/processors/silver_processor.py:14

↓ 1 callersClassStorageClient

S3/MinIO storage client.

src/utils/storage.py:11

ClassBaseProcessor

Abstract base class for all processors. Enforces interface contract and handles common operations. Design Pattern: Interface Inherit

src/processors/base_processor.py:10

Sales transaction schema.

src/models/schemas.py:7