Django-RQ

Django integration with RQ, a Redis based Python queuing library. Django-RQ is a simple app that allows you to configure your queues in Django's settings.py and easily use them in your project.

Support Django-RQ

If you find django-rq useful, please consider supporting its development via Tidelift.

Requirements

• Django (3.2+)
• RQ

Installation

• Install django-rq (or download from PyPI):

pip install django-rq

• Add django_rq to INSTALLED_APPS in settings.py:

INSTALLED_APPS = (
    # other apps
    "django_rq",
)

• Configure your queues in Django's settings.py:

RQ_QUEUES = {
    'default': {
        'HOST': 'localhost',
        'PORT': 6379,
        'DB': 0,
        'USERNAME': 'some-user',
        'PASSWORD': 'some-password',
        'DEFAULT_TIMEOUT': 360,
        'DEFAULT_RESULT_TTL': 800,
        'REDIS_CLIENT_KWARGS': {    # Eventual additional Redis connection arguments
            'ssl_cert_reqs': None,
        },
    },
    'with-sentinel': {
        'SENTINELS': [('localhost', 26736), ('localhost', 26737)],
        'MASTER_NAME': 'redismaster',
        'DB': 0,
        # Redis username/password
        'USERNAME': 'redis-user',
        'PASSWORD': 'secret',
        'SOCKET_TIMEOUT': 0.3,
        'CONNECTION_KWARGS': {  # Eventual additional Redis connection arguments
            'ssl': True
        },
        'SENTINEL_KWARGS': {    # Eventual Sentinel connection arguments
            # If Sentinel also has auth, username/password can be passed here
            'username': 'sentinel-user',
            'password': 'secret',
        },
    },
    'high': {
        'URL': os.getenv('REDISTOGO_URL', 'redis://localhost:6379/0'),  # If you're on Heroku
        'DEFAULT_TIMEOUT': 500,
    },
    'low': {
        'HOST': 'localhost',
        'PORT': 6379,
        'DB': 0,
    }
}

RQ_EXCEPTION_HANDLERS = ['path.to.my.handler']  # If you need custom exception handlers

Admin Integration

New in Version 4.0

Django-RQ automatically integrates with Django's admin interface. Once installed, navigate to /admin/django_rq/dashboard/ to access: - Queue statistics and monitoring dashboard - Job registry browsers (scheduled, started, finished, failed, deferred) - Worker management - Prometheus metrics endpoint (if prometheus_client is installed)

The views are automatically registered in Django admin and a link to the dashboard is added to the admin interface's sidebar. If you want to disable this link, add RQ_SHOW_ADMIN_LINK = False in settings.py.

Standalone URLs (Alternative)

For advanced use cases, you can also include Django-RQ views at a custom URL prefix:

# urls.py
urlpatterns += [
    path('django-rq/', include('django_rq.urls'))
]

This makes views accessible at /django-rq/ instead of within the admin interface at /admin/django_rq/dashboard/.

Template URL Resolution

Templates in django-rq use a custom {% rq_url %} tag to resolve view names inside either the admin integration or standalone URLs. The tag detects the current admin namespace (when present) and falls back to the django_rq: namespace, avoiding hard-coded prefixes and keeping links working in both modes.

If you copy or override django-rq templates, load the tag library and use rq_url instead of url for django-rq views:

{% load django_rq %}
<a href="https://github.com/rq/django-rq/raw/v4.1/{% rq_url 'home' %}">Django RQ</a>

Usage

Putting jobs in the queue

Django-RQ allows you to easily put jobs into any of the queues defined in settings.py. It comes with a few utility functions:

• enqueue - push a job to the default queue:

import django_rq
django_rq.enqueue(func, foo, bar=baz)

• get_queue - returns a Queue instance.

import django_rq
queue = django_rq.get_queue('high')
queue.enqueue(func, foo, bar=baz)

In addition to name argument, get_queue also accepts default_timeout, is_async, commit_mode, connection and queue_class arguments. For example:

queue = django_rq.get_queue('default', commit_mode='on_db_commit', is_async=True, default_timeout=360)
queue.enqueue(func, foo, bar=baz)

You can provide your own singleton Redis connection object to this function so that it will not create a new connection object for each queue definition. This will help you limit number of connections to Redis server. For example:

import django_rq
import redis

redis_cursor = redis.StrictRedis(host='', port='', db='', password='')
high_queue = django_rq.get_queue('high', connection=redis_cursor)
low_queue = django_rq.get_queue('low', connection=redis_cursor)

• get_connection - accepts a single queue name argument (defaults to "default") and returns a connection to the queue's Redis server:

import django_rq

redis_conn = django_rq.get_connection('high')

• get_worker - accepts optional queue names and returns a new RQ Worker instance for specified queues (or default queue):

import django_rq

worker = django_rq.get_worker()  # Returns a worker for "default" queue
worker.work()
worker = django_rq.get_worker('low', 'high')  # Returns a worker for "low" and "high"

@job decorator

To easily turn a callable into an RQ task, you can also use the @job decorator that comes with django_rq:

from django_rq import job

@job
def long_running_func():
    pass

long_running_func.delay()  # Enqueue function in "default" queue

@job('high')
def long_running_func():
    pass

long_running_func.delay()  # Enqueue function in "high" queue

You can pass in any arguments that RQ's job decorator accepts:

@job('default', timeout=3600)
def long_running_func():
    pass

long_running_func.delay()  # Enqueue function with a timeout of 3600 seconds.

It's possible to specify default for result_ttl decorator keyword argument via DEFAULT_RESULT_TTL setting:

RQ = {
    'DEFAULT_RESULT_TTL': 5000,
}

With this setting, job decorator will set result_ttl to 5000 unless it's specified explicitly or included in the queue config.

Running workers

django_rq provides a management command that starts a worker for every queue specified as arguments:

python manage.py rqworker high default low

If you want to run rqworker in burst mode, you can pass in the --burst flag:

python manage.py rqworker high default low --burst

If you need to use custom worker, job or queue classes, it is best to use global settings (see Custom queue classes and Custom job and worker classes). However, it is also possible to override such settings with command line options as follows.

To use a custom worker class, you can pass in the --worker-class flag with the path to your worker:

python manage.py rqworker high default low --worker-class 'path.to.GeventWorker'

To use a custom queue class, you can pass in the --queue-class flag with the path to your queue class:

python manage.py rqworker high default low --queue-class 'path.to.CustomQueue'

To use a custom job class, provide the --job-class flag.

Starting from version 2.10, running RQ's worker-pool is also supported:

python manage.py rqworker-pool default low medium --num-workers 4

Support for Scheduled Jobs

With RQ 1.2.0 you can use the built-in scheduler for your jobs. For example:

from datetime import datetime
from django_rq.queues import get_queue

queue = get_queue('default')
job = queue.enqueue_at(datetime(2020, 10, 10), func)

If you are using built-in scheduler you have to start workers with scheduler support:

python manage.py rqworker --with-scheduler

Support for RQ's CronScheduler

Create a cron configuration file:

# cron_config.py
from rq import cron
from myapp.tasks import send_report, sync_data

cron.register(send_report, queue_name='default', cron='0 9 * * *')  # Daily at 9:00 AM
cron.register(sync_data, queue_name='high', interval=30)  # Every 30 seconds

Then start the cron scheduler:

python manage.py rqcron cron_config.py

For more options, visit RQ's CronScheduler documentation.

Support for django-redis and django-redis-cache

If you have django-redis or django-redis-cache installed, you can instruct django_rq to use the same connection information from your Redis cache. This has two advantages: it's DRY and it takes advantage of any optimization that may be going on in your cache setup (like using connection pooling or Hiredis).

To configure it, use a dict with the key USE_REDIS_CACHE pointing to the name of the desired cache in your RQ_QUEUES dict. It goes without saying that the chosen cache must exist and use the Redis backend. See your respective Redis cache package docs for configuration instructions. It's also important to point out that since the django-redis-cache ShardedClient splits the cache over multiple Redis connections, it does not work.

Here is an example settings fragment for django-redis:

CACHES = {
    'redis-cache': {
        'BACKEND': 'redis_cache.cache.RedisCache',
        'LOCATION': 'localhost:6379:1',
        'OPTIONS': {
            'CLIENT_CLASS': 'django_redis.client.DefaultClient',
            'MAX_ENTRIES': 5000,
        },
    },
}

RQ_QUEUES = {
    'high': {
        'USE_REDIS_CACHE': 'redis-cache',
    },
    'low': {
        'USE_REDIS_CACHE': 'redis-cache',
    },
}

Suspending and Resuming Workers

Sometimes you may want to suspend RQ to prevent it from processing new jobs. A classic example is during the initial phase of a deployment script or in advance of putting your site into maintenance mode. This is particularly helpful when you have jobs that are relatively long-running and might otherwise be forcibly killed during the deploy.

The suspend command stops workers on all queues (in a single Redis database) from picking up new jobs. However currently running jobs will continue until completion.

# Suspend indefinitely
python manage.py rqsuspend

# Suspend for a specific duration (in seconds) then automatically
# resume work again.
python manage.py rqsuspend -d 600

# Resume work again.
python manage.py rqresume

Queue Statistics

Changed in Version 3.0

Various queue statistics are also available in JSON format via /django-rq/stats.json, which is accessible via a bearer token authentication scheme (defined in settings.py as RQ_API_TOKEN). Then, include the token in the Authorization header as a Bearer token: Authorization: Bearer <token> and access it via /django-rq/stats.json.

Note: Statistics of scheduled jobs display jobs from RQ built-in scheduler, not optional RQ scheduler.

Additionally, these statistics are also accessible from the command line.

python manage.py rqstats
python manage.py rqstats --interval=1  # Refreshes every second
python manage.py rqstats --json  # Output as JSON
python manage.py rqstats --yaml  # Output as YAML

Configuring Prometheus

django_rq also provides a Prometheus compatible view, which can be enabled by installing prometheus_client or installing the extra "prometheus-metrics" (pip install django-rq[prometheus]). The metrics are exposed at /django-rq/metrics/ and the following is an example of the metrics that are exported:

# HELP rq_workers RQ workers
# TYPE rq_workers gauge
# HELP rq_job_successful_total RQ successful job count
# TYPE rq_job_successful_total counter
# HELP rq_job_failed_total RQ failed job count
# TYPE rq_job_failed_total counter
# HELP rq_working_seconds_total RQ total working time
# TYPE rq_working_seconds_total counter
# HELP rq_jobs RQ jobs by status
# TYPE rq_jobs gauge
rq_jobs{queue="default",status="queued"} 0.0
rq_jobs{queue="default",status="started"} 0.0
rq_jobs{queue="default",status="finished"} 0.0
rq_jobs{queue="default",status="failed"} 0.0
rq_jobs{queue="default",status="deferred"} 0.0
rq_jobs{queue="default",status="scheduled"} 0.0

If you need to access this view via other HTTP clients (for monitoring purposes), you can define RQ_API_TOKEN. Then, include the token in the Authorization header as a Bearer token: Authorization: Bearer <token> and access it via /django-rq/metrics.

Configuring Logging

RQ uses Python's logging, this means you can easily configure rqworker's logging mechanism in Django's settings.py. For example:

```python LOGGING = { "version": 1, "disable_existing_loggers": False, "formatters": { "rq_console": { "format": "%(asctime)s %(message)s", "datefmt": "%H:%M:%S", }, }, "handlers": { "rq_console": { "level": "DEBUG", "class": "rq.logutils.ColorizingStreamHandler", "formatter": "rq_console", "exclude": ["%(asctime)s"], }, }, 'loggers': { "rq.worker": { "handlers": ["rq_console", "sentry"], "level": "DEBUG" },