Healthchecks

Healthchecks is a cron job monitoring service. It listens for HTTP requests and email messages ("pings") from your cron jobs and scheduled tasks ("checks"). When a ping does not arrive on time, Healthchecks sends out alerts.

Healthchecks comes with a web dashboard, API, 25+ integrations for delivering notifications, monthly email reports, WebAuthn 2FA support, team management features: projects, team members, read-only access.

The building blocks are:

• Python 3.12+
• Django 6.0
• PostgreSQL, MySQL or MariaDB

Healthchecks is licensed under the BSD 3-clause license.

Healthchecks is available as a hosted service at https://healthchecks.io/.

A Dockerfile and pre-built Docker images are available.

Screenshots:

The "My Checks" screen. Shows the status of all your cron jobs in a live-updating dashboard.

Each check has configurable Period and Grace Time parameters. Period is the expected time between pings. Grace Time specifies how long to wait before sending out alerts when a job is running late.

Alternatively, you can define the expected schedules using a cron expressions. Healthchecks uses the cronsim library to parse and evaluate cron expressions.

Check details page, with a live-updating event log.

Healthchecks provides status badges with public but hard-to-guess URLs. You can use them in your READMEs, dashboards, or status pages.

Setting Up for Development

If you are planning to developing Healthchecks, please read CONTRIBUTING.md.

To set up Healthchecks development environment:

• Install dependencies (Debian/Ubuntu):

sh sudo apt update sudo apt install -y gcc python3-dev python3-venv libpq-dev libcurl4-openssl-dev libssl-dev

• Prepare directory for project code and virtualenv. Feel free to use a different location:

sh mkdir -p ~/webapps cd ~/webapps

• Prepare virtual environment (with virtualenv you get pip, we'll use it soon to install requirements):

sh python3 -m venv .venv source .venv/bin/activate pip3 install wheel # make sure wheel is installed in the venv

• Check out project code:

sh git clone https://github.com/healthchecks/healthchecks.git

• Install requirements (Django, ...) into virtualenv:

sh pip install -r healthchecks/requirements.txt -r healthchecks/requirements-dev.txt

• macOS only - pycurl needs to be reinstalled using the following method (assumes OpenSSL was installed using brew):

sh export PYCURL_VERSION=`cat requirements.txt | grep pycurl | cut -d '=' -f3` export OPENSSL_LOCATION=`brew --prefix openssl` export PYCURL_SSL_LIBRARY=openssl export LDFLAGS=-L$OPENSSL_LOCATION/lib export CPPFLAGS=-I$OPENSSL_LOCATION/include pip uninstall -y pycurl pip install pycurl==$PYCURL_VERSION --compile --no-cache-dir

• Create database tables and a superuser account:

sh cd ~/webapps/healthchecks ./manage.py migrate ./manage.py createsuperuser

With the default configuration, Healthchecks stores data in a SQLite file hc.sqlite in the checkout directory (~/webapps/healthchecks).

• Run tests:

sh ./manage.py test

• Run development server:

sh ./manage.py runserver

The site should now be running at http://localhost:8000. To access Django administration site, log in as a superuser, then visit http://localhost:8000/admin/

Configuration

Healthchecks reads configuration from environment variables. See the full list of configuration parameters you can set via environment variables.

In addition, Healthchecks reads settings from the hc/local_settings.py file if it exists. You can set or override any standard Django setting in this file. You can copy the provided hc/local_settings.py.example as hc/local_settings.py and use it as a starting point.

If a setting is specified both as environment variable and in hc/local_settings.py, the latter takes precedence.

Accessing Administration Panel

Healthchecks comes with Django's administration panel where you can perform administrative tasks: delete user accounts, change passwords, increase limits for specific users, inspect contents of database tables.

To access the administration panel,

• if you haven't already, create a superuser account: ./manage.py createsuperuser
• log into the site using superuser credentials
• in the top navigation, "Account" dropdown, select "Site Administration"

Sending Emails

Healthchecks must be able to send email messages, so it can send out login links and alerts to users. Specify your SMTP credentials using the following environment variables:

• Implicit TLS (recommended): python DEFAULT_FROM_EMAIL = "valid-sender-address@example.org" EMAIL_HOST = "your-smtp-server-here.com" EMAIL_PORT = 465 EMAIL_HOST_USER = "smtp-username" EMAIL_HOST_PASSWORD = "smtp-password" EMAIL_USE_TLS = False EMAIL_USE_SSL = True

  Port 465 should be the preferred method according to RFC8314 Section 3.3: Implicit TLS for SMTP Submission. Be sure to use a TLS certificate and not an SSL one.

• Explicit TLS: python DEFAULT_FROM_EMAIL = "valid-sender-address@example.org" EMAIL_HOST = "your-smtp-server-here.com" EMAIL_PORT = 587 EMAIL_HOST_USER = "smtp-username" EMAIL_HOST_PASSWORD = "smtp-password" EMAIL_USE_TLS = True

For more information, have a look at Django documentation, Sending Email section.

Receiving Emails

Healthchecks comes with a smtpd management command, which starts up a SMTP listener service. With the command running, you can ping your checks by sending email messages to your-uuid-here@my-monitoring-project.com email addresses.

Start the SMTP listener on port 2525:

./manage.py smtpd --port 2525

Send a test email:

curl --url 'smtp://127.0.0.1:2525' \
    --mail-from 'foo@example.org' \
    --mail-rcpt '11111111-1111-1111-1111-111111111111@my-monitoring-project.com' \
    -F '='

Sending Alerts and Reports

Healthchecks comes with a sendalerts management command, which continuously polls database for any checks changing state, and sends out notifications as needed. Within an activated virtualenv, you can manually run the sendalerts command like so:

./manage.py sendalerts

In a production setup, you will want to run this command from a process manager like systemd or supervisor.

Healthchecks also comes with a sendreports management command which sends out monthly reports, weekly reports, and the daily or hourly reminders.

Run sendreports without arguments to run any due reports and reminders and then exit:

./manage.py sendreports

Run it with the --loop argument to make it run continuously:

./manage.py sendreports --loop

Database Cleanup

Healthchecks deletes old entries from api_ping, api_flip, and api_notification tables automatically. By default, Healthchecks keeps the 100 most recent pings for every check. You can set the limit higher to keep a longer history: go to the Administration Panel, look up user's Profile and modify its "Ping log limit" field.

Healthchecks also provides management commands for cleaning up auth_user (user accounts) and api_tokenbucket (rate limiting records) tables, and for removing stale objects from external object storage.

• Remove user accounts that are older than 1 month and have never logged in:

sh ./manage.py pruneusers

• Remove old records from the api_tokenbucket table. The TokenBucket model is used for rate-limiting login attempts and similar operations. Any records older than one day can be safely removed.

sh ./manage.py prunetokenbucket

• Remove old objects from external object storage. When an user removes a check, removes a project, or closes their account, Healthchecks does not remove the associated objects from the external object storage on the fly. Instead, you should run pruneobjects occasionally (for example, once a month). This command first takes an inventory of all checks in the database, and then iterates over top-level keys in the object storage bucket, and deletes any that don't also exist in the database.

sh ./manage.py pruneobjects

When you first try these commands on your data, it is a good idea to test them on a copy of your database, not on the live database right away. In a production setup, you should also have regular, automated database backups set up.

Two-factor Authentication

Healthchecks optionally supports two-factor authentication using the WebAuthn standard. To enable WebAuthn support, set the RP_ID (relying party identifier ) setting to a non-null value. Set its value to your site's domain without scheme and without port. For example, if your site runs on https://my-hc.example.org, set RP_ID to my-hc.example.org.

External Authentication

Healthchecks supports external authentication by means of HTTP headers set by reverse proxies or the WSGI server. This allows you to integrate it into your existing authentication system (e.g., LDAP or OAuth) via an authenticating proxy. When this option is enabled, healthchecks will trust the header's value implicitly, so it is very important to ensure that attackers cannot set the value themselves (and thus impersonate any user). How to do this varies by your chosen proxy, but generally involves configuring it to strip out headers that normalize to the same name as the chosen identity header.

To enable this feature, set the REMOTE_USER_HEADER value to a header you wish to authenticate with. HTTP headers will be prefixed with HTTP_ and have any dashes converted to underscores. Headers without that prefix can be set by the WSGI server itself only, which is more secure.

When REMOTE_USER_HEADER is set, Healthchecks will: - assume the header contains user's email address - look up and automatically log in the user with a matching email address - automatically create an user account if it does not exist - disable the default authentication methods (login link to email, password)

The header name in REMOTE_USER_HEADER must be specified in upper-case, with any dashes replaced with underscores, and prefixed with HTTP_. For example, if your authentication proxy sets a X-Authenticated-User request header, you should set REMOTE_USER_HEADER=HTTP_X_AUTHENTICATED_USER.

Note on using local_settings.py: When Healthchecks reads settings from environment variables and encounters the REMOTE_USER_HEADER environment variable, it sets two settings, REMOTE_USER_HEADER and AUTHENTICATION_BACKENDS. This logic has already run by the time Healthchecks reads local_settings.py. Therefore, if you configure Healthchecks using the local_settings.py file instead of environment variables, and specify REMOTE_USER_HEADER there, you will also need a line which sets the other setting, AUTHENTICATION_BACKENDS:

REMOTE_USER_HEADER = "HTTP_X_AUTHENTICATED_USER"
AUTHENTICATION_BACKENDS = ["hc.accounts.backends.CustomHeaderBackend"]

External Object Storage

Healthchecks can optionally store large ping bodies in S3-compatible object storage. To enable this feature, you will need to:

• ensure you have the MinIO Python library installed:

bash pip install minio * configure the credentials for accessing object storage: S3_ACCESS_KEY, S3_SECRET_KEY, S3_ENDPOINT, S3_REGION and S3_BUCKET.

Healthchecks will use external object storage for storing any request bodies that exceed 100 bytes. If the size of a request body is 100 bytes or below, Healthchecks will still store it in the database.

Healthchecks automatically removes old stored ping bodies from object storage while uploading new data. However, Healthchecks does not automatically clean up data when you delete checks, projects or entire user accounts. Use the pruneobjects management command to remove data for checks that don't exist any more.

When external object storage is not enabled (the credentials for accessing object storage are not set), Healthchecks stores all ping bodies in the database. If you enable external object storage, Healthchecks will still be able to access the ping bodies already stored in the database. You don't need to migrate them to the object storage. On the other hand, if you later decide to disable external object storage, Healthchecks will not have access to the externally stored ping bodies any more. And there is currently no script or management command for migrating ping bodies from external object storage back