Fetch INSPIRE BBR data and enrich with GraphQL API. Strategy: 1. Download and process INSPIRE BBR GPKG 2. Extract bu

Mock implementation of fetch_raw_soap_response. Executes a SOAP operation and returns the raw response.

Create a DriveFile from a Google Drive API response. Args: file_data: Google Drive API response for a file parent_pat

Mock generate_deterministic_track_id function.

Generate field analysis PMTiles for a specific year. Args: year: Target year Returns: Path to generated PMTi

Generate PMTiles for a specific year and resolution.

Generate a timestamp string in the format YYYYMMDD_HHMMSS. Returns: Timestamp string

Geocode multiple addresses by ID. Args: address_ids: List of DAWA address IDs max_concurrent: Maximum number

* Get maximum points allowed for zoom level and layer type

* Get a PMTiles URL for an optional layer and return null when the file is * missing so map consumers can skip the source entirely.

Get A_over and A_under for a crop using actual formulas. Args: crop_code: FVM crop code yield_kg_ts_ha: Harvested yield in k

Create a mock aiohttp session.

Create a mock aiohttp session.

Find date-related columns dynamically with improved logic for animal welfare data.

Get the expected payment rate for a scheme. Args: ordning_kode: Standardized scheme code use_derived: If True, use data-deri

Mock get file size.

Get a configured incremental processor instance.

Get FVM credentials from environment variables (legacy simple auth).

Get current memory usage statistics. Returns: dict: Memory usage information including DuckDB and system memory

Get performance statistics including cache hit rates. Returns: Dictionary with performance metrics

Get the schema of a DuckDB table.

Get application settings.

Get all pipeline sources that should run at the specified frequency. Args: frequency: The schedule frequency to filter by Retur

Get a storage manager instance. Args: storage_type: Type of storage ('r2' or 'local') bucket_name: R2 bucket name (required for R

Fetch table metadata from the API

Mock get_vetstat_credentials function.

Import pipeline context data from a previous job.

Check if a herd is in the high-volume herds list.

Yield every record from the WP REST `kontrol` index, across all pages.

Mock list files.

Load and integrate all field-related data for a given year. Args: year: Target year Returns: DuckDB table na

Fetches animal movement data for a given herd using besListAktOms. Args: chr_dyr_client: The CHR_dyr SOAP client username: U

Load field data from FVM silver layer for a specific CVR and year. This queries the agricultural field data including: - Fie

Load all transportation data sources dynamically using cloud storage patterns. Uses unified pipeline pattern: shared DuckDB connection with S

Log error message if logger available.

Log detailed geometry statistics for a DuckDB table. This method analyzes geometry properties including dimensions, area, ve

Migrate the common _save_data pattern to StorageAccess. This replaces the pattern: ```python # OLD with tempfile.NamedTemporaryF

Resolve an R2 glob to sorted r2:// URIs.

Build an R2 URI for the configured bucket.

Build a path without a medallion layer prefix.

Load parquet file(s) from R2 prefix into a DuckDB table. Returns row count.

Run silver processing for DMI monthly data. Args: bronze_data: Optional data from bronze stage. If None,

Run multi-year kommune-level PFAS analysis from cloud storage data.

Save metadata about the downloaded file.

Save metadata alongside data file

Save schema information to storage in JSON format.

Save table to parquet file (legacy method).

Select the primary address from a list of addresses. Args: addresses: List of address dictionaries Returns:

Set the source CRS of the data being processed. Bronze layer implementations should call this when they determine the CRS of

Set up logging configuration. Args: log_level: Logging level (DEBUG, INFO, WARNING, ERROR)

Setup rclone configuration for Cloudflare R2. Returns: True if successful, False otherwise

Generate SQL to transform geometry for final Supabase upload. This should be the ONLY place where data is transformed to WGS84. Use at t

Stream contents of a temp file one item at a time.

Load JSON, sanitize nested arrays to remove empty keys, flatten nested arrays, and return as a Pandas DataFrame.

* Update viewport and recalculate bounds

Upload file to storage using s3fs.

Validate that all pipeline dependencies are properly configured. Returns: List of validation error messages (empty if all valid)

Validate that a field ID follows expected FVM field ID formats. FVM field IDs can be: - Simple numeric: "8", "23", "6", "2"

Create element.

Initialize the validator.

Apply schema to a processed file. Args: output_path: Path to the processed file metadata: File metadata s

Bootstrap confidence interval for point-biserial correlation. Returns (ci_low, ci_high) at the (1-alpha) level. More appropriate than Fisher

Build the SELECT SQL for clean dataset transform. Args: group_filter: Optional GROUP value to filter on (e.g. 'pest').

Build the SQL query for field analysis data export. Args: table_name: Name of the integrated data table year: Target