Parses dbt yaml file to get db credentials. Also substitutes environment variables. Parameters ---------- config_file: DbParamsConfig

Sets up a custom logger which logs to file as well as console Parameters ---------- log_name : str Location of log file, eg './lo

Creates the self tests' schema and runs the queries in `additional_query` if provided Parameters ----------

Gets project-dependent path of `models-paths` (i.e. where DBT models are located) from dbt_project.yml config file items Parameter

Helper to run sub-process, sending output to logger. Parameters ---------- cmd : str Command to be run by subprocess, eg 'ls'

Internal function - checks if the path exists Parameters ---------- path : str path of the file Retu

Gets the full row of configured_tests fields by finding the corresponding record in dot.configured_tests Parameters ---------- test_t

Obtains the db connection for the self tests db Parameters ---------- mock_get_filename_safely connection: D

Writes a config file from a template

Figures out test type from the dbt logs INTERNAL function, do not use Parameters ---------- node: dict json from dbt man

formats the list of uuids if equals to a string of {'id1', 'id2'} gives back a list of ['id1', 'id2'] Parameters ---------- uuid

Get schema suffix for test objects generated by dbt Parameters ---------- dot_config : Optional[dict]

Generates a test_results dataframe using the test_summary dataframe and associated test results DB views Parameters ----------

Read generated json files. Assumes the cleanup has run and most recent results are in _archive files TODO maybe dbt output _test.json is

Gets key from dbt_project.yml config file items Converts the result from list to str, assuming the list has only one element Param

Drops the self tests' schema Parameters ---------- schema: str schema for self tests conn: Optio

Get sample test summary DF + run id Returns ------- Tuple[pd.DataFrame, uuid.UUID] test_summary

Figures out parameters from each of the tests of the dbt output rows Parameters ---------- row: dict json from dbt logs & ma

Cleans up schema from self_tests_dot to dot

Gets original (i.e. project_independent) path of `key` (can be anything, but refers to either dbt models or tests) from dbt_project.yml confi

Figures out test parameters from the dbt logs INTERNAL function, do not use Parameters ---------- node: dict json from d

Regardless of the parameters for any of the GE tests in db, some extra parameters for config need to be added,e.g. schema in which the core m

Archived previous dbt results Parameters ---------- logger : logger Logger

Reads configured_entities from the DOT schema and writes them as sql files for dbt Parameters ---------- schema_dot: str DOT

Generates custom dbt models Parameters ---------- project_id : str Project id for run, eg 'Muso' logger : logger Logg

Default configs for creating engine Parameters ---------- db_credentials : dict db credentials dictiontay Returns --

Takes a dbt config file and replaces any environment variable syntax with the environment variable. Syntax looks like this ... pass: "{{ env_

Function to parse GE csv results file (that was created by parse_results) to form a test_summaries standard dataframe. Parameters

Generates a test_results dataframe, one row per row of data tested Parameters ---------- id_column_values : list List of values f

Function to generate dbt test yml files and Great Expectation .json files based on the contents of database table dot.configured_tests. Par

Gets project-dependent path of `tests-paths` (i.e. where DBT tests are located) from dbt_project.yml config file items Parameters

Gets queries from file Parameters ---------- f: file file object schema: str schema

Figures out a human readable test_name, and a shorten version that can become a view name Parameters ---------- node : dict D

Generates test UUID based on configured_test fields by finding the corresponding record in dot.configured_tests Parameters ----------

Figures out the view definition of a failing test Parameters ---------- project_id: str e.g. Muso, for obtaining credentials

Reads config file safely Returns ------- config: str content of config file

Loads credentials and transforms password for project id and connection Parameters ---------- project_id : str

Converts a pandas Series or a string representation into a list, or pass on the list if it already was one. This is to make sure we can read the r

Remove extra paramets added by `add_ge_schema_parameters` so that test parameters correspond to the ones stored in the DOT configuration data

Runs the main set of dbt tests as defined (generated from db) in dbt/core/*.yml Parameters ---------- logger : logger Logger

Runs dbt custom sql tests Parameters ---------- logger : logger Logger

Wrapper around the DOT pipeline which will set status, start and end times in dot.run_status. Also catches exception and updates dot.run_log t

Saves test results to the DOT DB into dot.test_results. Parameters ---------- test_rows : Pandas dataframe Dataframe of test resu

Internal function - gets credentials either for the project or the DOT Parameters ---------- db_config : dict

Adapts core entities definition to point the schema Jinja statement to the project schema Parameters ---------- schema_project:

check outputs recursively for dbt logs

Function to generate project_directories if they don't exist. Creates ... |dot | | config | | |<project_id> | | | | dbt

Creates a failed tests view for Great Expectations tests Parameters ---------- project_id: str Current run project ID

setup for dbt tests - dbt_project config file - entities to be tested

Sets configuration to determine how the DAG will run from Airflow config. Used if called didn't provide configuration. Returns file handle.

Takes a dbt config file and replaces any environment variable syntax with the environment variable. Syntax looks like this ... pass: "{{ env_

Function to parse GE csv results file (that was created by parse_results) to form a test_summaries standard dataframe. Parameters -

Generates test coverage report Parameters ---------- logger : logger Logger project_id : str Project ID, eg 'Muso'. M

Generates a test_results dataframe, one row per row of data tested Parameters ---------- failing_ids : list List of unique ids in

Function to generate master configuration files using the entries in config/dot_config.yml and also to set up required output directories (per pr

Gets the metadata local object to this module Returns ------- MetaData

Extracts data from object in source Postgres DB and saves to target DOT database in data schema. Parameters ---------- object_name_

Runs a subprocess, but captures output to the logger Parameters ---------- pipe : Popen pipe Subprocess POpen object, for example

Creates a csv file parsed from great expectations results json object Parameters ---------- results: Pandas dataframe Returns

Refresh ddbb metadata. Slow operation! Can be called from anywhere and refreshes the object local object to this module @param engine: e

Runs all the actions for dbt

Runs the full pipeline of DOT: - dbt tests - great expectation tests - report generation and save of results to the database Paramete

Runs Great Expectations tests Parameters ---------- logger : logger Logger project_id : str Project ID, eg 'Muso'. Mu

Saves data to target DOT database in data schema. Parameters ---------- object_name_in: String The Postgres db object to syn

Generates tests summary stats for total, failed and passed test rows Parameters ---------- tests_summary : pandas datafram Test s

Overrides password with environment variable if db_config has environment variable syntax Parameters ---------- db_config : json

Gets entity name from entity_id Parameters ---------- project_id Returns -------

We are syncing new data where new columns and columns types might change. Postgres will prevent ALTER TABLE if any views exist, so we will dr

custom expectation

Compares distributions of measurements across CHWs to detect ouliers. This expectation produces warnings rather than pointing out errors due t

Fake expectation for test purpuses. The test takes the first 5 rows of the entity and considers them failed rows Paramet

Gets original (i.e. project_independent) path of `model-paths` (i.e. where DBT models are located) from dbt_project.yml config file items

Gets original (i.e. project_independent) path of `test-paths` (i.e. where DBT tests are located) from dbt_project.yml config file items

notebooks/MT - Immunization Exploration Patients are given OPV 1, 2, or 3 immunizations too early Parameters -------

Mock paths of config files Parameters ---------- path Returns -------

Runs chv-specific tests TODO: Needs more work, chv tests were important in previous projects. Check run_everything as this function might be

Method to run the DOT.

creates DB schema for the demo dataset by default

Extracts data from object in source Postgres DB and saves to target DOT database in data schema. Parameters ---------- object_name_

drops the DB schema for the demo dataset by default

test function add_ge_schema_parameters

test a possible_duplicate_forms w missing parameters

test get_dbt_config_custom_schema_output_objects

test output df generated from dbt results in json format (dbt target directory)

Formats `uuid_list` from postgres as actually a list Returns -------

test yaml file creation for 1 core entity -see file in filename below

test yaml file creation for 1 core entity -see file in filename below