This repo contains the installer and quickstart setup for the DataKitchen Data Observability product suite. * DataOps TestGen is a data quality verification tool that does five main tasks: (1) data profiling, (2) new dataset screening and hygiene review, (3) algorithmic generation of data quality validation tests, (4) ongoing production testing of new data refreshes and (5) continuous periodic monitoring of datasets for anomalies. * DataOps Observability monitors every tool used in the journey of data from data source to customer value, from any team development environment into production, across every tool, team, data set, environment, and project so that problems are detected, localized, and understood immediately.
| Software | Tested Versions | Command to check version |
|---|---|---|
| Requirements for TestGen & Observability | ||
| Python |
Most Linux and macOS systems have Python pre-installed.
On Windows machines, you will need to download and install it. | 3.9, 3.10, 3.11, 3.12 | python3 --version |
| Docker
Docker Compose (pre-installed with Docker Desktop) | 25.0.3
2.24.6 | docker -v
docker compose version |
| Additional Requirements for Observability |
| Minikube | 1.32.0 | minikube version |
| Helm | 3.13.3, 3.14.3 | helm version |
| Minikube Driver
0.20210107
25.0.3 |
hyperkit -v
docker -v |
On Unix-based operating systems, use the following command to download it to the current directory. We recommend creating a new, empty directory.
curl -o dk-installer.py 'https://raw.githubusercontent.com/DataKitchen/dk-installer/main/dk-installer.py'
dk-installer.py file from this repo. python3 dk-installer.py --help or python3 dk-installer.py <command> --help.Until we make the images public, Docker credentials with access to the datakitchen namespace have to be configured on your machine.
export DOCKER_USERNAME=<username>export DOCKER_PASSWORD=<password>The Data Observability quickstart walks you through Dataops Observability and TestGen capabilities to demonstrate how our products cover critical use cases for data and analytic teams.
Before going through the quickstart, complete the prequisites above and then the following steps to install the two products and setup the demo data. For any of the commands, you can view additional options by appending --help at the end.
The installation downloads the latest Docker images for TestGen and deploys a new Docker Compose application. The process may take 5~10 minutes depending on your machine and network connection.
python3 dk-installer.py tg install
Once it completes, verify that you can login to the UI with the URL and credentials provided in the output.
The installation downloads the latest Helm charts and Docker images for Observability and deploys the application on a new minikube cluster. The process may take 5~30 minutes depending on your machine and network connection.
python3 dk-installer.py obs install
For Windows and macOS running M-series (ARM) chip only: The Docker driver is not allowed to expose the HTTP ports to the host machine, so the following command has to be run after the install to access the application. Leave this process running, and continue the next steps on another terminal window.
python3 dk-installer.py obs expose
Verify that you can login to the UI with the URL and credentials provided in the output.
The demo-config.json file generated by the Observability installation must be present in the folder.
python3 dk-installer.py tg run-demo --export
In the TestGen UI, you will see that new data profiling and test results have been generated. Additionally, in the Observavility UI, you will see that new test outcome events have been received.
The demo-config.json file generated by the Observability installation must be present in the folder.
python3 dk-installer.py obs run-demo
In the Observability UI, you will see that new journeys and events have been generated.
The demo-config.json file generated by the Observability installation must be present in the folder.
python3 dk-installer.py obs run-heartbeat-demo
In the Observability UI, you will see that new agents have been generated on the Integrations page.
Leave this process running, and continue with the quickstart guide to tour the applications.
The docker compose CLI can be used to operate the installed TestGen application. All commands must be run in the same folder that contains the docker-compose.yaml file generated by the installation.
Access the testgen CLI: docker compose exec engine bash (use exit to return to the regular terminal)
Stop the app: docker compose down
Restart the app: docker compose up
The minikube and kubectl command line tools can be used to operate the Observability application.
Inspect the pods: kubectl get pods
Get pod logs: kubectl logs <POD ID>
Stop the app: minikube stop
Restart the app: minikube start
After completing the quickstart, you can remove the demo data from the applications with the following steps.
Stop the process that is running the Agent Heartbeat demo using Ctrl + C.
Note: Currently, the agents generated by the heartbeat demo are not cleaned up.
The demo-config.json file generated by the Observability installation must be present in the folder.
python3 dk-installer.py tg delete-demo
python3 dk-installer.py obs delete-demo
python3 dk-installer.py tg delete
python3 dk-installer.py obs delete
$ claude mcp add data-observability-installer \
-- python -m otcore.mcp_server <graph>