Split 'Indskærpelse (2 = Indskærpelse)' into ('Indskærpelse', 2). Handles four observed formats: 1. 'Ingen anmærkninger (2 = Indskærpelse)'

Read silver data from a specific storage path into a DuckDB table. Args: dataset_name: Name of the dataset for logging

Replicate silver's DuckDB SQL transform without pyarrow. Silver does: pa.Table.from_pylist(data) → DuckDB UNNEST → DataFrame. We repl

Save intermediate results to avoid memory issues.

Save a raw .rds file to storage. Path structure: bronze/{dataset}/{run_timestamp}/{filename} Args: rds_content:

Determine if files should be uploaded to R2. Returns: True if upload should proceed, False otherwise

Download all GeoDanmark buildings in batches with progress tracking. This method continuously fetches batches until no more data is

Beregner korrektionsfaktor for afgrøderester i græs (k_græs). Args: n_graes_kg_n_ha (float): Mængden af N afsat under afgræsning [kg

Calculate NH3 emissions from pig manure management. Formula: NH3 (kg) = antal_dyr × EF_nh3 Where: - antal_dyr: Number o

Classify a de minimis scheme code. Args: ordning: Ordning code from de minimis data Returns: Tuple of (ordning_kode, or

Classify an FVM commitment type code. Args: tilsagnstype: Tilsagnstype code from FVM data (e.g., "36", "67") Returns: T

Collect all field IDs and field UUIDs from the output data after processing completes. Returns: Set of field IDs found i

Create a from the raw WFS data using DuckDB. Args: raw_data (List[str]): List of raw WFS response strings l

Creates the herd_owners table including attributes from besaetning details. Args: con: The DuckDB connection bes_details_raw: Nam

Create a robust SOAP client with fallback.

Ensure directory exists (no-op for cloud storage, creates directory for local). Args: path: Directory path Raises:

Find the CVR column in a list of column names. Args: columns: List of column names Returns: Name of the CVR column, or

Generate deterministic UUID for any entity type with custom identifier. Args: entity_type: Type of entity (e.g., 'employ

Geocode an address using Datavask API with caching. Args: address: Full address string Returns: Dic

Get the number of animals of a specific type. Args: animal_type: Identifier for the animal type (e.g., 'jersey_conventio

Get the bronze directory path with appropriate suffixes for matrix job separation.

Get the final list of years to process based on configuration and available data. Args: available_years: Dictionary of available

* Check if point is within bounds

Join pesticide disaggregation with BMD data for PFAS detection.

Resolve the newest parquet in datasets shaped like `<prefix>/<timestamp>/<filename>`. This avoids deep recursive globs such as `foo_*/*/data.

Load FVM field data from cloud storage and prepare for spatial intersection.

Load emission factors from JSON file. Returns: dict: Emission factors organized by source type (manure_storage, housing_emissions, e

Load pesticide disaggregation data from cloud storage for a specific year.

Log info message if logger available.

Convenience function to lookup pig FE value. Args: pig_type: "årssøer", "smågrise", or "slagtesvin" production_system: "conv

Convert meters to approximate degrees for Denmark latitude. This is useful for ST_DWithin operations when working with EPSG:4326 data.

Process files from a Google Drive folder. Args: folder_id: ID of the Google Drive folder to process (deprecated, use drive_folder

Read metadata from a JSON file. Args: metadata_path: Path to the metadata file Returns: FileMetadata object

Resolve storage bucket from override, env vars, or default. Priority: override > STORAGE_BUCKET > R2_BUCKET > GCS_BUCKET > "landbruget-data"

Drop a table or view, handling both types safely.

Run the data source processing pipeline. This method must be implemented by all concrete source classes. It should handle th

Helper to generate SQL expression for cleaning string columns: - Trims whitespace - Treats empty strings as null Args: col_name:

Save a pandas DataFrame or a DuckDB table name to Parquet.

Generate SQL to transform geometry to processing CRS (EPSG:25832). Use this in Silver layer to normalize all data to EPSG:25832. Only ne

Generate SQL to transform geometry to Danish UTM for metric operations. Uses ``always_xy := true`` so EPSG:4326 is interpreted as (lon, lat)

Execute the full transformation process. Returns: Path to the processed Parquet file

Verify that SPATIAL_JOIN operator is being used in a query. Args: conn: DuckDB connection query: SQL query to analyze R

Build a bare bucket/key path from parts, stripping extra slashes.

Build the request body for the Kemidata download (CSV) API.

Build the per-field pesticide aggregation SQL. All per-field dosage values — both the BMD-classification detail strings (pfas/diquat/

Calculate NLES5 nitrogen washout estimates using the full model. Returns: Table name containing final NLES5 estimates

Clean up temporary files created during PMTiles generation for a specific year.

Clean up temporary files to prevent disk space issues.

Convert .rds file to pandas DataFrame using pyreadr. Args: rds_path: Path to the local .rds file Returns:

Create a standardized address record for geocoding. Args: addr: Original address data source_type: "company"

Write a tiny corrupt file to the fetcher's output directory.

Create a simple DuckDB table with test data.

Try downloading a batch with retries on transient errors.

Safely drop a table if it exists.

Fetch data from an ArcGIS REST API service. This method handles fetching geospatial data from ArcGIS REST API endpoints. It

Fetch geometries for a batch of building IDs using POST request to avoid URI length limits. Args: building_ids: List of

Filter addresses based on basic requirements. Args: addresses: List of address dictionaries Returns:

Find a column from a list of candidate names.

Get WFS GetFeature request parameters. Args: layer_name (str): Name of the layer to request get_count_only (

Load DST JSONSTAT data into DuckDB. Args: json_data (Dict[str, Any]): Raw JSONSTAT data from DST API table_n

Load Danish kommune boundaries from cloud storage using proper authentication.

Load data from cloud storage into a DuckDB table using optimized cloud storage access - WITH CLEANUP.

Log current memory usage to help monitor cleanup effectiveness. Args: context: Context description for the memory measur

Log a preview of the generated NLES5 estimates in a structured format.

Helper to create a client with geocoding disabled.

Helper to create a client with geocoding disabled.

Delegate to memory utils.

Parse coordinate string into list of (x, y) tuples. Args: coord_string (str): Space-separated coordinate values "x1 y1 x

Merge ``updates`` into the parquet file's schema KV metadata, preserving other keys. Reads the parquet file, merges ``updates`` on top of any exi

Process bronze GeoJSON payloads into a DuckDB spatial feature table.

Resolve geometry expression to EPSG:25832 using bounds-based CRS detection.

Run DMA scraper main.py with given args.

Extract and save CVR numbers discovered in the CHR pipeline silver data. Args: con: DuckDB connection silver_dir: Silver dat

Determine if an address should be geocoded, considering fallback to historical addresses ONLY for companies with zero current address