Iaso : a geospatial data management platform

Full documentation is at docs.openiaso.com

Introduction

Iaso is a georegistry and data collection web platform structured around trees of organization units (also known a master lists)

The main tasks it allows accomplishing are:

• Data collection using XLSForm forms linked to org units through a mobile application.
• Import, comparison and merging of multiple Org Units' trees, both through a web application and an API allowing manipulation through data science tools like Jupyter notebooks.
• Validation of received data for Org Units' trees and forms.
• Exports of the Org Units' trees and form data, either in csv, xlsx, GeoPackage or through an api.

Video presentation of IASO at FOSDEM 2021, with slides.

Long Intro

Iaso is a platform created to support geo-rich data collection efforts, mainly in public health in emerging countries. The key feature that it supports is that any survey is linked to an organizational unit that is part of a canonical hierarchy. Each one of these org. units can have a location and a territory. The mobile data collection tool can be used to enrich this hierarchy with additional GPS coordinates, names corrections, etc ... which can then be validated by officials of the organizations in question through the web dashboard. This leads to continuous improvements of the geographic references available through the routine activities already planned. e.g. locating and registering health facilities while investigating malaria cases.

The tool has been used in multiple data collection efforts, notably in the domain of Performance Based Financing of health services in D.R. Congo, Niger, Cameroon and Nigeria and is more and more used to compare multiple versions of official organisational hierarchies when a canonical one needs to be rebuilt (for example to rebuild a school map for DRC). To help for this type of project, we provide location selection interfaces, multiple levels of audits and an API open to data scientists for analysis and mass edits.

Iaso has been created by the company Bluesquare, specialised in software and services for public health, and has become open source under the MIT License in November 2020.

Iaso is made of a white labeled Android application using Java/Kotlin, reusing large parts of the ODK projects, and a web platform programmed using Python/GeoDjango on top of PostGIS. Frontend is mainly React/Leaflet. One of the aims is the ease of integration with other platforms. We already have csv and geopackage imports and exports and target easy integration with OSM.

Structure

This repository contains Iaso frontend and backend, respectively in Python Django and JS React. They interact via an API implemented via Django rest framework, all data is stored in Postgresql or the media/ directory.

A companion mobile app for Android allow submitting Form and creating org unit.

Form can also be filled in a web interface via the Enketo companion service. Both Iaso and Enketo need to be configured to work together. It is possible to run an Enketo service locally, see Enketo section below.

More documentation on the Front End part is present in hat/assets/README.rst

Data Model / Glossary

IASO inspired its terminology both from ODK and from DHIS2. We will highlight some equivalences that might help you.

This is not (yet) the complete Data Model, but here are the main concepts/model in Iaso:

• Iaso is multi tenant. Tenant are called and represented by the model Account. It represents roughly one client org or country. It also represents the natural limit of right for a user.
• Each Django's User has a linked Iaso Profile that link it to an Account and store extra parameters for the user.
• Each tenant can have multiple Project. Projects are linked to one android version App via the app_id. We use the link to control what a user can see from that app.
• DHIS2 is a standard server application and web UI in the industry to handle Health Data. Iaso can import and export data (forms and org unit) to it.
• OrgUnit (Organizational Unit) is a Node of the GeoRegistry tree. e.g a particular Country, City or Hospital. each belonging to each other via a parent relationship.
  • They can have a type OrgUnitType e.g. Country, City, Hospital
  • they can belong to multiple Group, e.g. Urban Region or Campaign 2017
  • DHIS2 has the concept of Group but not Type so when importing from a DHIS2 Instance all the type will be Unknown and OrgUnit will belong to group like Clinic
  • GroupSet are Group of group. Used when we export Group to DHIS2
  • OrgUnit may have a position in space, it can be an area, the geom field is then used, or just a Point, the location field is then used.
    • It's technically possible to have both
    • a OrgUnit may have no geographical info
    • a OrgUnit may geographically be outside its parent.
• DataSource links OrgUnit and Group imported from the same source, e.g a DHIS2 instance, a CSV or a GeoPackage.
  • A source_ref on the imported instance is used to keep the reference from the original source, so we can match it again in the future (when updating the import or exporting it back)
  • SourceVersion is used to keep each version separated. e.g. each time we import from DHIS2 we create a new version.
  • OrgUnit (for their parent) and Group should only reference other OrgUnit and Group in the same version. (This is not enforced everywhere yet)
• Task are asynchronous function that will be run by a background worker in production. eg: Importing Data from DHIS2. see Worker section below for more info.
• Form is the definition of a Form (list of question and their presentation).
  • The model contain the metadata, the actual definition is done in a XSLForm as an attached file.
  • Form are linked to one or more Project. This is used to know which Form are presented in the Mobile App.
  • Form can have multiple versions
• Instance or Form instance is the Submission of a form. A form that has actually been filed by a user.
  • Instance can be GeoTagged and/or linked to a OrgUnit
  • Note: We are moving to use Submission everywhere in the UI, but it is still in progress. please submit PR.
  • Submission cannot be done via the Iaso UI itself but through Enketo or the Mobile App.
• EntityType represents a type of person or object to which we want to attach multiple submissions to track said submissions in time and across OrgUnits.
• Entity represents an actual person or object, defined by its EntityType. A concrete example is given in the docstrings of iaos.models.entity
• APIImport are used to log some request from the mobile app, so we can replay them in case of error. See vector_control Readme
• audit.Modification are used to keep a history of modification on some models (mainly orgunit). See audit readme
• Link are used to match two OrgUnit (in different sources or not) that should be the same in the real world. Links have a confidence score indicating how much we trust that the two OrgUnit are actually the same.

They are usually generated via AlgorithmRun, or the matching is done in a Notebook and uploaded via the API.

Development environment

Setup

A running local instance for development can be spin up via docker compose which will install and configure all dep in separate container. As such your computer should only need:

• git
• docker
• docker compose

If docker compose give you trouble, make sure it can connect to the docker daemon.

If you use an Apple Silicon Mac, ensure export DOCKER_DEFAULT_PLATFORM=linux/amd64 is set.

A pgdata-iaso folder, containing the database data, will be created in the parent directory of the git repository.

1. Environment variables

The docker-compose.yml file contains sensible defaults for the Django application.

Other environment variables can be provided by a .env file.

As a starting point, you can copy the sample .env.example file and edit it to your needs.

cp .env.example .env

note

all the commands here need to be run in the project directory, where you cloned the repository

2. Build the containers

This will build and download the containers.

docker compose build

3. Start the database

docker compose up db

(if you get this message: "Database is uninitialized and superuser password is not specified. You must specify POSTGRES_PASSWORD" you can set POSTGRES_PASSWORD=postgres in the .env file )

4. Run migrations

In a separate bash (without closing yet the started db), launch the migrations

docker compose run --rm iaso manage migrate

(If you get a message saying that the database iaso does not exist, you can connect to your postgres instance using

psql -h localhost -p 5433 -U postgres

then type

create database iaso;

to create the missing database.)

(If you get a message saying that /opt/app/entrypoint.sh does not exist, you need to disable auto crlf from github, more details: https://stackoverflow.com/questions/38905135/why-wont-my-docker-entrypoint-sh-execute)

git config --global core.autocrlf input
git rm --cached -r .
git reset --hard

Then rebuild & migration

docker compose build
docker compose run --rm iaso manage migrate

5. Start the server

To start all the containers (backend, frontend, db)

docker compose up

If you get a message saying that entrypoint.sh cannot be find and working on Windows, the git repository has an entry point script with Unix line endings (\n). But when the repository was checked out on a windows machine, git decided to try and be clever and replace the line endings in the files with windows line endings (\r\n). The solution is to disable git's automatic conversion:

git config --global core.autocrlf input

Reset the repo (make sure you don't have unpushed changes):

git rm --cached -r .
git reset --hard

And then rebuild:

docker compose build

The web server should be reachable at http://localhost:8081 (you should see a login form).

The docker-compose.yml file describes the setup of the containers. See section below for a lit

6. Create a superuser

To log in to the app or the Django admin, a superuser needs to be created with:

docker compose exec iaso ./manage.py createsuperuser

You can now log in at the admin section: http://localhost:8081/admin.

Then additional users with custom groups and permissions can be added through the Django admin or loaded via fixtures.

7. Create and import data

To create the initial account, project and profile, do the following:

docker compose exec iaso ./manage.py create_and_import_data

And run the following command to populate your database with a tree of org units (these are childcare schools in the West of DRC):

docker compose exec iaso ./manage.py tree_importer \
    --org_unit_csv_file testdata/schools.csv \
    --data_dict testdata/data_dict.json \
    --source_name wb_schools_2019 \
    --version_number=1 \
    --project_id=1\
     --main_org_unit_name maternelle

You can now log in on http://localhost:8081

Alternatively to this step and following steps you can import data from DHIS2 see section below.

8. Create a form

Run the following command to create a form:

docker compose exec iaso ./manage.py create_form

At this point, if you want to edit forms directly on your machine using Enketo, go to the Enketo setup section of this README (down below).

Once you are done, you can click on the eye for your newly added form, click on "+ Create", tap a letter, then enter, select the org unit, then click "Create submission".

If Enketo is running and well setup, you can fill the form now.

9. Create other cool stuff

You can now start to develop additional features on Iaso!

10. Import OrgUnit, Forms and Submission from DHIS2

Alternatively or in addition to steps 7-8, you can import data from the DHIS2 demo server (play.dhis2.org).

First find a running version from play.dhis2.org. Take the most recent as the other ones may return a 404. Follow the link, eg: https://play.im.dhis2.org/stable-2-40-3-1 In this example the version is 2.40.3.1. Pass it to docker compose run:

In a new bash, run the command

docker compose run --rm iaso manage seed_test_data --mode=seed --dhis2version=2.40.3.1

The hierarchy of OrgUnit, group of OrgUnit, Forms, and their Submissions will be imported. Type of OrgUnit are not handled at the moment

you can then log in through http://127.0.0.1:8081/dashboard with :

• user : testemail2.37.7.1
• password: testemail2.37.7.1

11. Activating the Polio plugin (optional)

Set the PLUGINS environment variable to polio. You can do so by adding the following line in your root .env:

PLUGINS=polio

Run commands inside the docker

Each docker container uses the entrypoint.