Deepchecks - Continuous Validation for AI & ML: Testing, CI & Monitoring

Deepchecks is a holistic open-source solution for all of your AI & ML validation needs, enabling you to thoroughly test your data and models from research to production.

  <img alt="Deepchecks continuous validation parts." src="https://github.com/deepchecks/deepchecks/raw/0.19.1/docs/source/_static/images//readme/deepchecks_continuous_validation_light.png">

  👋 Join Slack   |   📖 Documentation   |   🌐 Blog   |  
🐦 Twitter  

🧩 Components

Deepchecks includes: - Deepchecks Testing (Quickstart, docs): - Running built-in & your own custom Checks and Suites for Tabular, NLP & CV validation (open source). - CI & Testing Management (Quickstart, docs): - Collaborating over test results and iterating efficiently until model is production-ready and can be deployed (open source & managed offering). - Deepchecks Monitoring (Quickstart, docs): - Tracking and validating your deployed models behavior when in production (open source & managed offering).

This repo is our main repo as all components use the deepchecks checks in their core. See the Getting Started section for more information about installation and quickstarts for each of the components. If you want to see deepchecks monitoring's code, you can check out the deepchecks/monitoring repo.

⏩ Getting Started

  <h3>
     💻 Installation
  </h3>

Deepchecks Testing (and CI) Installation

pip install deepchecks -U --user

For installing the nlp / vision submodules or with conda: - For NLP: Replace deepchecks with "deepchecks[nlp]", and optionally install alsodeepchecks[nlp-properties] - For Computer Vision: Replace deepchecks with "deepchecks[vision]". - For installing with conda, similarly use: conda install -c conda-forge deepchecks.

Check out the full installation instructions for deepchecks testing here.

Deepchecks Monitoring Installation

To use deepchecks for production monitoring, you can either use our SaaS service, or deploy a local instance in one line on Linux/MacOS (Windows is WIP!) with Docker. Create a new directory for the installation files, open a terminal within that directory and run the following:

pip install deepchecks-installer
deepchecks-installer install-monitoring

This will automatically download the necessary dependencies, run the installation process and then start the application locally.

The installation will take a few minutes. Then you can open the deployment url (default is http://localhost), and start the system onboarding. Check out the full monitoring open source installation & quickstart.

Note that the open source product is built such that each deployment supports monitoring of a single model.

🏃‍♀️ Quickstarts

  <h4>
     Deepchecks Testing Quickstart
  </h4>

Jump right into the respective quickstart docs:

• Tabular Quickstarts
• NLP Quickstarts
• Vision Quickstarts

to have it up and running on your data.

Inside the quickstarts, you'll see how to create the relevant deepchecks object for holding your data and metadata (Dataset, TextData or VisionData, corresponding to the data type), and run a Suite or Check. The code snippet for running it will look something like the following, depending on the chosen Suite or Check.

from deepchecks.tabular.suites import model_evaluation
suite = model_evaluation()
suite_result = suite.run(train_dataset=train_dataset, test_dataset=test_dataset, model=model)
suite_result.save_as_html() # replace this with suite_result.show() or suite_result.show_in_window() to see results inline or in window
# or suite_result.results[0].value with the relevant check index to process the check result's values in python

The output will be a report that enables you to inspect the status and results of the chosen checks:

  <h4>
     Deepchecks Monitoring Quickstart
  </h4>

Jump right into the open source monitoring quickstart docs to have it up and running on your data. You'll then be able to see the checks results over time, set alerts, and interact with the dynamic deepchecks UI that looks like this:

  <h4>
     Deepchecks CI & Testing Management Quickstart
  </h4>

Deepchecks managed CI & Testing management is currently in closed preview. Book a demo for more information about the offering.

For building and maintaining your own CI process while utilizing Deepchecks Testing for it, check out our docs for Using Deepchecks in CI/CD.

🧮 How does it work?

At its core, deepchecks includes a wide variety of built-in Checks, for testing all types of data and model related issues. These checks are implemented for various models and data types (Tabular, NLP, Vision), and can easily be customized and expanded.

The check results can be used to automatically make informed decisions about your model's production-readiness, and for monitoring it over time in production. The check results can be examined with visual reports (by saving them to an HTML file, or seeing them in Jupyter), processed with code (using their pythonic / json output), and inspected and collaborated on with Deepchecks' dynamic UI (for examining test results and for production monitoring).

  <h2>
     ✅ Deepchecks' Core: The Checks
  </h2>

• All of the Checks and the framework for customizing them are implemented inside the Deepchecks Testing Python package (this repo).
• Each check tests for a specific potential problem. Deepchecks has many pre-implemented checks for finding issues with the model's performance (e.g. identifying weak segments), data distribution (e.g. detect drifts or leakages) and data integrity (e.g. find conflicting labels).
• Customizable: each check has many configurable parameters, and custom checks can easily be implemented.
• Can be run manually (during research) or triggered automatically (in CI processes or production monitoring)
• Check results can be consumed by:
• Visual output report - Saving to HTML(result.save_to_html('output_report_name.html')) or viewing them in Jupyter (result.show()).
• Processing with code - with python using the check result's value attribute, or saving a JSON output
• Deepchecks' UI - for dynamic inspection and collaboration (of test results and production monitoring)
• Optional conditions can be added and customized, to automatically validate check results, with a a pass ✓, fail ✖ or warning ! status
• An ordered list of checks (with optional conditions) can be run together in a "Suite" (and the output is a concluding report of all checks that ran)

📜 Open Source vs Paid

Deepchecks' projects (deepchecks/deepchecks & deepchecks/monitoring) are open source and are released under AGPL 3.0.

The only exception are the Deepchecks Monitoring components (in the deepchecks/monitoring repo), that are under the (backend/deepchecks_monitoring/ee) directory, that are subject to a commercial license (see the license here). That directory isn't used by default, and is packaged as part of the deepchecks monitoring repository simply to support upgrading to the commercial edition without downtime.

Enabling premium features (contained in the backend/deepchecks_monitoring/ee directory) with a self-hosted instance requires a Deepchecks license. To learn more, book a demo or see our pricing page.

Looking for a 💯% open-source solution for deepcheck monitoring? Check out the Monitoring OSS repository, which is purged of all proprietary code and features.

👭 Community, Contributing, Docs & Support

Deepchecks is an open source solution. We are committe