Remove duplicate rows using DuckDB SQL.

Mock download JSON.

Drop a table or view if it exists.

OPTIMAL: Export DuckDB table directly to cloud storage without DataFrame conversion. This provides maximum performance: - No

Create and save metadata. Call at pipeline end. Args: record_count: Number of records processed. file_size_bytes: Siz

Get a reference to the current data buffer.

Determine if an HTTP status code should trigger a retry. Args: status_code: HTTP status code Returns: True if the request sh

List files matching pattern with their timestamps. Returns list of tuples: (file_path, timestamp) Useful for finding the mos

Mock implementation of load_herd_list. Fetches a list of herd numbers for a given species and usage code.

Prepare geometries for spatial operations with coordinate system alignment.

✅ OPTIMIZED: Save DuckDB table directly to cloud storage without DataFrame conversion. This method provides maximum performance by:

Set up DuckDB with test data directly (synchronous, no cloud storage access). This is a test helper that bypasses the async _setup_duckdb me

Add an error to the validation result. Args: result: ValidationResult to update error: Error message to add

Clean and standardize string values from XML. This method converts values to strings and removes leading/trailing whitespace.

Find the latest bronze directory and CSV file.

Normalize enums and special characters using DuckDB SQL.

Register data (DataFrame, etc.) as a DuckDB table.

STRATEGY 1: THE MAIN WORKHORSE - Area matching with 2% tolerance (92% success rate!) WHAT THIS STRATEGY DOES: ==============

Fetch a chunk of features with rate limiting and retries

Get the latest silver data directory path for worker safety. The silver path historically used a space in the dataset name, but the c

Process raw XML data into a DuckDB table. This method parses XML data containing wetland features and converts it to a DuckD

Beregner N2O udledning fra gødning og CO2e. Args: n_total_kg_ha (float): Total N i gødningstypen pr. ha [kg N/ha]. areal_ha

Provide CVR validation function to tests. Returns: Callable that validates CVR format.

Check if a file exists. Args: path: File path Returns: True if file exists, False otherwise

Format the log record with colors.

Get the CAP pillar for a scheme (1 = EAGF, 2 = EAFRD). Args: ordning_kode: Standardized scheme code Returns: 1 for Pill

Convert empty strings to null values using DuckDB SQL.

Check if an exception should trigger a retry. Args: exception: The exception to check Returns: True if the operation should

Load CSV data using DuckDB.

Write GeoJSON file from a DuckDB query result. Args: duckdb_conn: DuckDB connection query: SQL query that returns geo

Clean up intermediate tables for a specific year to free memory - AGGRESSIVE for GitHub Actions.

Create a dissolved (merged) version of the wetlands dataset using DuckDB-spatial. CRITICAL: This method works on the original coordi

Find the latest file matching a pattern using the existing cloud storage utility. Args: storage_access: StorageAccess instance

Format municipality name from ALL CAPITALS to proper Title Case. Args: municipality_name: Municipality name (often in AL

Generate WFS request parameters. This method creates a dictionary of parameters needed for a WFS GetFeature request, includi

Generate unique table name with proper lifecycle management.

Load a JSON file from the package data directory.

Mark a table as protected from cleanup.

Run point-biserial correlation for a single PFAS substance vs intensity measure. Returns a dict with results or None if insufficient data.

Enhanced save method with pipeline metadata for data tracing. This method combines the existing _save_data functionality with automa

Add a warning to the validation result. Args: result: ValidationResult to update warning: Warning message to add

Creates the herds table (excluding owner/user identifiers) from besaetning details. Args: con: The DuckDB connection bes_details_

Get display name for a substance.

Check if file exists.

Write buffered data to consolidated files with memory-efficient processing.

Generate PMTiles from GeoJSON using tippecanoe. Args: geojson_path: Path to input GeoJSON file output_path: Path for

Main function to process CHR data from bronze to silver. This function now supports both streaming (memory-efficient) and legacy (in-memory) mode

Execute the stage processing.

Run the PMTiles generation pipeline. Args: target_year: Optional specific year to process Returns: Dictionar

Save metadata to a JSON file. Args: metadata: FileMetadata object target_path: Path to the file the metadata is for

Serialize element to string.

Validate geometry CRS before transformation. Samples geometries from a table and validates they match the expected CRS. Raises ValueErro

Validate that a CVR string matches the required format. Args: value: CVR string to validate Returns: True if valid 8-di

Fetch a chunk of features with retry logic. This method retrieves a batch of features from the WFS service starting at the specified

Return column reference or NULL if column doesn't exist.

Get the disaggregated results.

Cast columns to appropriate types using DuckDB SQL.

Provide CHR validation function to tests. Returns: Callable that validates CHR format.

Classify an EU scheme name to standardized code. Args: scheme_name: Scheme name from støtteoplysninger "Scheme" column Returns:

Delete old timestamped versions, keeping only the most recent *keep*. Expects the standard medallion layout where each version is stored unde

* Escapes CSV field values (handles commas, quotes, newlines)

ULTIMATE PERFORMANCE: Export table directly to cloud storage using native DuckDB access. This bypasses all temporary files and strea

Fetches data using Playwright automation. Returns list of (filter_name, csv_bytes).

* Get a PMTiles URL with caching optimization

Create a mock aiohttp session.

Get information about a table.

Load missing env vars from Google Cloud Secret Manager. Args: project: GCP project ID. Defaults to GCP_PROJECT env var. Returns

Check if a scheme name indicates a summary/total row. These rows must be filtered out to avoid double-counting. Args: scheme_na

Check if a single coordinate pair is in UTM range for Denmark. Args: x: X coordinate (easting) y: Y coordinate (northing)

Check if a single coordinate pair is in WGS84 range for Denmark. Args: x: X coordinate (longitude) y: Y coordinate (latitude

Mock load_vetstat_antibiotics function.

Set context for logging. Args: **kwargs: Key-value pairs to add to the logging context

Setup DuckDB with required extensions - OPTIMIZED for GitHub Actions.

Mock upload from DuckDB table.

Verify coordinate order in a geometry table. Returns: - bool: True if coordinates are in LON/LAT order - str

Aggressive cleanup for GitHub Actions constraints - only used at end of processing.

Replicate silver's column name cleaning logic.

Clean up temporary files and directories.

Combine all intermediate files into final GeoParquet.

Get integer environment variable.