Verified as a Digital Public Good by the Digital Public Goods Alliance.

Sample data

You are free to use these rich datasets ranging from global COVID-19 patients to United States childhood obesity records, from 1000 records to over a million patients, as well as a synthetic dataset showcasing DOT’s capabilities on frontline health data.

The Data Observation Toolkit (DOT)

The Data Observation Toolkit (DOT) can be used to monitor data in order to flag problems with data integrity and scenarios that might need attention. Typical tests include checks for missing/duplicate and inconsistent data, outliers, and domain-specific signals such as a missed follow-up medical treatment after initial diagnosis.

DOT includes a simple user interface for configuring the powerful DBT and Great Expectations libraries, as well as a database for storing and classifying data monitoring results.

The ultimate aim of DOT is to provide easier access to data monitoring without the need for deep technical skills.

In 2019, the United Nations Statistical Commission stated: “Every misclassified or unrecorded death is a lost opportunity to ensure other mothers and babies do not die in the same way. When it comes to health, better data can be a matter of life and death.” DataKind developed DOT in response to a demand for tool to increase trust in public health data which underpins the ability of a health system to provide equitable, data-driven, optimized health service delivery and improve policy response.

DOT was developed with our valuable and globally distributed frontline health partners working to strengthen frontline health systems, expertise from Ministries of Health, frontline health workers and funders.

Read more here:

Pathways to Increasing Trust in Public Health Data

Empowering Health Worker and Community Health Systems: Data Integrity and the Future of Intelligent Community Health Systems in Uganda

Harnessing the power of data science in healthcare

How Data Empowers Health Workers—and Powers Health Systems

Quick start

The fastest way to get started using DOT is to use the Docker environment and demo data provided.

Prerequisites

1. You will need to install Docker desktop. First make sure you have checked the Docker prerequisites. We recommend using at least 4GB memory which can be set in the docker preferences, but this can vary depending on the volume of data being tested
2. If running on a Mac M1/M2 chip, install Rosetta and set export DOCKER_DEFAULT_PLATFORM=linux/amd64 in the terminal where you will run the instructions below

Setup

You will need Python and some modules installed to run the DOT quick-start demo:

1. Install Python 3.8.9

2. Install the necessary python packages by running the following commands in your terminal (Additional information Mac/Linux terminal, additional information Windows terminal):

3. pip install gdown

4. pip install python-on-whales

Alternatively, you can use the provided environment.yml if you have miniconda installed.

Running the DOT quick-start demo environment

1. Start Docker
2. Open a Terminal and cd into the docker sub-folder of where you installed DOT
   • Example Windows: cd C:<PATH TO WHERE YOU INSTALLED DOT>\Data-Observation-Toolkit\docker
   • Example Mac/Linux: cd <PATH TO WHERE YOU INSTALLED DOT>/Data-Observation-Toolkit/docker/
3. When on Mac/Linux, run python3 run_demo.py or when on Windows, run python3 .\run_demo.py
4. The script will start DOT and open a browser with the DOT User Interface. See demo video below for how to get started.
5. Once you are done with the demo, press return in your Terminal, to stop the docker containers

Video Demo of DOT

Note: Be sure to activate audio ...

https://user-images.githubusercontent.com/8402586/195226567-fe035544-7075-4750-8bd8-ddfa7f57a811.mp4

The not-so-quick start, setting up DOT

Building the DOT Docker environment

If you wish to build DOT yourself ...

1. Save a postgres password to your local environment export POSTGRES_PASSWORD=<some password you decide>
2. Open a terminal and change directory into the docker sub-directory of DOT: cd ./docker
3. Build the docker containers: docker compose build
4. Start them: docker compose up -d

Your environment is now running!

Running DOT

1. docker exec -it dot /bin/bash
2. cd dot
3. python3 ./run_everything.py --project_id 'ScanProject1'

This will run the sample tests against the demo data and save results to the DOT database.

Setting up the DOT User Interface

The above steps will start the DOT user interface, which will allow you to manage DOT and see results. You need to do a few steps the first time you use it ...

1. Go to http://localhost:82/
2. Click the button and register to create a login (keep note of the password)
3. Once created, click 'Build my own' on the next popup
4. Click app smith icon top-left to go back tom homepage
5. Top-right next to the new button, click on the '...' and select import
6. Select Import from file and navigate to file ./docker/appsmith/DOT App V2.json
7. You will be prompted to enter details for the database connection. You can set these as required, but if using the DOT dockerized Postgres database, the parameters are:
   • Host address: dot-db
   • Port: 5432
   • Database name: dot_db
   • Authentication > User name: postgres
   • Authentication > Password: <THE PASSWORD YOU USED WHEN BUILDING DOT>

You should now see the DOT user interface in developer mode (ie you could edit it). To run in end-user mode:

1. Click button top-right click the 'Deploy' button. This should open a new tab with the user interface in all its glory!

Note: If you want to remove Appsmith information on the deployed app, add ?embed=True at the end of the deployed app URL.

Viewing test results

Your test results will be in the dot-db container. You can view the results by opening a shell in the dot-db container:

docker exec -it dot-db /bin/bash

Then running the psql client locally in that container:

psql -U postgres -d dot_db

Or if you prefer a database client (like DBeaver), you can use their settings:

host=localhost
port=5433 
database=dot_db
user=postgres
password=<the POSTGRES_PASSWORD you set above>

Note: The host and port are set in the docker-compose.yml

To see some raw results you can run SELECT * from dot.test_results LIMIT 100;. Some more advanced queries are provided below.

Some useful DOT database queries

The following queries might be useful for visualizing the DOT DB schema and data:

Statistics on failed tests

SELECT
   tr.run_id,
   ct.test_type,
   COUNT(*)
FROM
   dot.test_results tr,
   dot.configured_tests ct 
WHERE
   tr.test_id = ct.test_id
GROUP BY
   tr.run_id,
   ct.test_type
ORDER BY
   ct.test_type

Same, but adding in test description and entity names in grouping ...

SELECT
   tr.run_id,
   tr.view_name, 
   ct.test_type,
   ct.description,
   ce.entity_name,
   COUNT(*)
FROM
   dot.test_results tr,
   dot.configured_tests ct,
   dot.configured_entities ce 
WHERE
   tr.test_id = ct.test_id AND
   ce.entity_id = ct.entity_id
GROUP BY
   tr.run_id,
   tr.view_name, 
   ct.test_type,
   ct.description,
   ce.entity_name
ORDER BY
   tr.run_id,
   tr.view_name, 
   ct.test_type,
   ct.description,
   ce.entity_name

Linking DOT Data scenarios with configured tests

SELECT 
   s.*,
   ct.*
FROM
   dot.scenarios s,
   dot.configured_tests ct 
WHERE 
   s.scenario_id=ct.scenario_id

Linking configured tests to test results

SELECT 
   s.*,
   ct.*,
   ce.entity_name,
   tr.*
FROM
   dot.scenarios s,
   dot.configured_tests ct,
   dot.test_results_summary tr,
   dot.configured_entities ce
WHERE 
   s.scenario_id=ct.scenario_id AND
   tr.test_id=ct.test_id AND
   ce.entity_id = ct.entity_id
LIMIT 10

Deactivating all tests except one dbt and one great expectation

update dot.configured_tests set test_activated=false where test_id not in ('7db8742b-c20b-3060-93e2-614e35da2d4b','0f26d515-a70f-3758-8266-8da326d90eb6')

Seeing the raw data for failed tests

dot.test_results column view_name provides the name of the DB view which holds the data for failed tests. Additionally, columns id_column_name and id_column_value provide the columns to match in the DB entity view, ie the data that was tested. Finally, you can query to get the underlying data for each test using function get_dot_data_record

SELECT 
   tr.test_id,
   tr.status,
   dot.get_test_result_data_record(ce.entity_name, tr.id_column_name, 
   tr.id_column_value,'public_tests')
FROM
   dot.scenarios s,
   dot.configured_tests ct,
   dot.configured_entities ce,
   dot.test_results tr
WHERE 
   s.scenario_id=ct.scenario_id AND
   tr.test_id=ct.test_id and 
   ce.entity_id=ct.entity_id
LIMIT 10

Where the function parameters are:

• Test entity name
• Test result ID column name (in entity view)
• Test Result ID column value
• Test results schema name

This returns a json record for the data that was tested. Note: If using the airflow environment, change public_tests to the schema where the data is, for example data_flights_db.

Configuring DOT

The following sections provide instructions for adding entities and tests to DOT directly in the DOT database. You can also use the DOT user interface for tests, for more details please see section see section The DOT User Interface. If you want to test DOT with health related data, please feel free to use this [fabricated dataset] (https://docs.google.com/spreadsheets/d/1l6inpa6ykgUewC-MJkgrQwwc8HjUiTLLJFEDBtAMDB4/edit#gid=31188808) which resembles appointment information and contains some quality issues which DOT can highlight.

How to add new entities

The DOT will run tests against user-defined views onto the underlying data. These views are called "entities" and defined in table dot.configured_entities:

Column	Description
entity_id	Name of the entity e.g. ancview_danger_sign
entity_category	Category of the entity e.g. anc => needs to be in dot.entity_categories
entity_definition	String for the SQL query that defines the entity

For example, this would be an insert command to create ancview_danger_sign:

INSERT INTO dot.configured_entities (project_id,entity_id,entity_category,entity_definition,date_added,date_modified,last_updated_by) VALUES('Project1', 
'ancview_danger_sign', 'anc', '{{ config(materialized=''view'') }}
{% set schema = <schema> %}

select *
from {{ schema }}.ancview_danger_sign');

All entities use Jinja macro statements - the parts between { ... } - which the DOT uses to create the entity materialized views in the correct database location. Use the above format for any new entities you create.

The SQL for the view definition can be more complex than the example above, combining data from multiple underlying tables or views. For example:

{{ config(materialized=''view'') }}
{% set schema = <schema> %}

select ap.*,
        ap.lmp as lmp_date,
        DATE_PART(''day'', reported - lmp) as days_since_lmp
from {{ schema }}.ancview_pregnancy ap

When you add a new entity to the configuration, take a look to the existing dot.entity_categories to associate your new entity to one of them. If you need to add a new category, the table dot.entity_categories has the following columns: | Column | Description | | :----------- | :----------- | | entity_category | Category of the entity e.g. anc | | description | A description of the category |

An example of an insert statement would be: INSERT INTO dot.entity_categories VALUES('anc', 'antenatal care');

How to configure tests

Tests are defined in the table dot.configured_tests. Each test has an associated test_type, a list of which can be found in table dot.test_types (see section "DOT Database Schema" below for more details on the full schema).

To use one of these test types for a new test, insert a new row in dot.configured_tests.

Here are the columns included in a test: | Column | Description | | :----------- | :----------- | test_activated | Whether the test is activa