github.com/OptimalBits/bull @v4.16.5 sqlite

The fastest, most reliable, Redis-based queue for Node.


Carefully written for rock solid stability and atomicity.









<a href="#-sponsors-"><strong>Sponsors</strong></a> ·
<a href="#bull-features"><strong>Features</strong></a> ·
<a href="#uis"><strong>UIs</strong></a> ·
<a href="#install"><strong>Install</strong></a> ·
<a href="#quick-guide"><strong>Quick Guide</strong></a> ·
<a href="#documentation"><strong>Documentation</strong></a>

Check the new Guide!

<a href="https://gitter.im/OptimalBits/bull">
  <img src="https://badges.gitter.im/Join%20Chat.svg"/>
</a>
<a href="https://join.slack.com/t/bullmq/shared_invite/zt-1nbtpk6mv-TItWpF9jf3k4yrCaS0PPZA">
  <img src="https://img.shields.io/badge/Slack-4A154B"/>
</a>
<a href="http://badge.fury.io/js/bull">
  <img src="https://badge.fury.io/js/bull.svg"/>
</a>
<a href="https://coveralls.io/github/OptimalBits/bull?branch=master">
  <img src="https://coveralls.io/repos/github/OptimalBits/bull/badge.svg?branch=master"/>
</a>
<a href="http://isitmaintained.com/project/OptimalBits/bull">
  <img src="http://isitmaintained.com/badge/open/optimalbits/bull.svg"/>
</a>
<a href="http://isitmaintained.com/project/OptimalBits/bull">
  <img src="http://isitmaintained.com/badge/resolution/optimalbits/bull.svg"/>
</a>
    <a href="https://twitter.com/manast">
  <img src="https://img.shields.io/twitter/follow/manast?label=Stay%20updated&style=social"/>
</a>

🚀 Sponsors 🚀

Dragonfly is a new Redis™ drop-in replacement that is fully compatible with BullMQ and brings some important advantages over Redis™ such as massive better performance by utilizing all CPU cores available and faster and more memory efficient data structures. Read more here on how to use it with BullMQ.

📻 News and updates

Bull is currently in maintenance mode, we are only fixing bugs. For new features check BullMQ, a modern rewritten implementation in Typescript. You are still very welcome to use Bull if it suits your needs, which is a safe, battle tested library.

Follow me on Twitter for other important news and updates.

🛠 Tutorials

You can find tutorials and news in this blog: https://blog.taskforce.sh/

Used by

Bull is popular among large and small organizations, like the following ones:

Official FrontEnd

Supercharge your queues with a professional front end: - Get a complete overview of all your queues. - Inspect jobs, search, retry, or promote delayed jobs. - Metrics and statistics. - and many more features.

Sign up at Taskforce.sh

Bull Features

• [x] Minimal CPU usage due to a polling-free design.
• [x] Robust design based on Redis.
• [x] Delayed jobs.
• [x] Schedule and repeat jobs according to a cron specification.
• [x] Rate limiter for jobs.
• [x] Retries.
• [x] Priority.
• [x] Concurrency.
• [x] Pause/resume—globally or locally.
• [x] Multiple job types per queue.
• [x] Threaded (sandboxed) processing functions.
• [x] Automatic recovery from process crashes.

And coming up on the roadmap...

• [ ] Job completion acknowledgement (you can use the message queue pattern in the meantime).
• [ ] Parent-child jobs relationships.

UIs

There are a few third-party UIs that you can use for monitoring:

BullMQ

• Taskforce

Bull v3

• Taskforce
• bull-board
• bull-repl
• bull-monitor
• Monitoro

Bull <= v2

• Matador
• react-bull
• Toureiro

Monitoring & Alerting

• With Prometheus Bull Queue Exporter

Feature Comparison

Since there are a few job queue solutions, here is a table comparing them:

Install

npm install bull --save

or

yarn add bull

Requirements: Bull requires a Redis version greater than or equal to 2.8.18.

Typescript Definitions

npm install @types/bull --save-dev

yarn add --dev @types/bull

Definitions are currently maintained in the DefinitelyTyped repo.

Contributing

We welcome all types of contributions, either code fixes, new features or doc improvements. Code formatting is enforced by prettier. For commits please follow conventional commits convention. All code must pass lint rules and test suites before it can be merged into develop.

Quick Guide

Basic Usage

const Queue = require('bull');

const videoQueue = new Queue('video transcoding', 'redis://127.0.0.1:6379');
const audioQueue = new Queue('audio transcoding', { redis: { port: 6379, host: '127.0.0.1', password: 'foobared' } }); // Specify Redis connection using object
const imageQueue = new Queue('image transcoding');
const pdfQueue = new Queue('pdf transcoding');

videoQueue.process(function (job, done) {

  // job.data contains the custom data passed when the job was created
  // job.id contains id of this job.

  // transcode video asynchronously and report progress
  job.progress(42);

  // call done when finished
  done();

  // or give an error if error
  done(new Error('error transcoding'));

  // or pass it a result
  done(null, { framerate: 29.5 /* etc... */ });

  // If the job throws an unhandled exception it is also handled correctly
  throw new Error('some unexpected error');
});

audioQueue.process(function (job, done) {
  // transcode audio asynchronously and report progress
  job.progress(42);

  // call done when finished
  done();

  // or give an error if error
  done(new Error('error transcoding'));

  // or pass it a result
  done(null, { samplerate: 48000 /* etc... */ });

  // If the job throws an unhandled exception it is also handled correctly
  throw new Error('some unexpected error');
});

imageQueue.process(function (job, done) {
  // transcode image asynchronously and report progress
  job.progress(42);

  // call done when finished
  done();

  // or give an error if error
  done(new Error('error transcoding'));

  // or pass it a result
  done(null, { width: 1280, height: 720 /* etc... */ });

  // If the job throws an unhandled exception it is also handled correctly
  throw new Error('some unexpected error');
});

pdfQueue.process(function (job) {
  // Processors can also return promises instead of using the done callback
  return pdfAsyncProcessor();
});

videoQueue.add({ video: 'http://example.com/video1.mov' });
audioQueue.add({ audio: 'http://example.com/audio1.mp3' });
imageQueue.add({ image: 'http://example.com/image1.tiff' });

Using promises

Alternatively, you can return promises instead of using the done callback:

videoQueue.process(function (job) { // don't forget to remove the done callback!
  // Simply return a promise
  return fetchVideo(job.data.url).then(transcodeVideo);

  // Handles promise rejection
  return Promise.reject(new Error('error transcoding'));

  // Passes the value the promise is resolved with to the "completed" event
  return Promise.resolve({ framerate: 29.5 /* etc... */ });

  // If the job throws an unhandled exception it is also handled correctly
  throw new Error('some unexpected error');
  // same as
  return Promise.reject(new Error('some unexpected error'));
});

Separate processes

The process function can also be run in a separate process. This has several advantages: - The process is sandboxed so if it crashes it does not affect the worker. - You can run blocking code without affecting the queue (jobs will not stall). - Much better utilization of multi-core CPUs. - Less connections to redis.

In order to use this feature just create a separate file with the processor:

// processor.js
module.exports = function (job) {
  // Do some heavy work

  return Promise.resolve(result);
}

And define the processor like this:

// Single process:
queue.process('/path/to/my/processor.js');

// You can use concurrency as well:
queue.process(5, '/path/to/my/processor.js');

// and named processors:
queue.process('my processor', 5, '/path/to/my/processor.js');

Repeated jobs

A job can be added to a queue and processed repeatedly according to a cron specification:

```js paymentsQueue.process(function (job) { /