elsdoerfer/celery-exporter repository overview

⁠celery-exporter

⁠Table of Contents

• Why another exporter?⁠
  • Features⁠
• Usage⁠
  • Enable events using the CLI⁠
  • Running the exporter⁠
• Metrics⁠

⁠Why another exporter?

While I was adding Celery monitoring to a client site I realized that the existing brokers either didn't work, exposed incorrect metric values or didn't expose the metrics I needed. So I wrote this exporter which essentially wraps the built-in Celery monitoring API and exposes all of the event metrics to Prometheus in real-time.

⁠Features

• Uses the built in real-time monitoring component in Celery⁠ to expose Prometheus metrics
• Tracks task status (task-started, task-succeeded, task-failed etc)
• Tracks which workers are running and the number of active tasks
• Follows the Prometheus exporter best practises⁠
• Works with both Redis and RabbitMQ
• Deployed as a Docker image or Python single-file binary (via PyInstaller)
• Exposes a health check endpoint at /health
• Grafana dashboards provided by the Celery-mixin
• Prometheus alerts provided by the Celery-mixin

⁠Dashboards and alerts

Alerting rules can be found here⁠. By default we alert if:

• A task failed in the last 10 minutes.
• No Celery workers are online.

Tweak these to suit your use-case.

The Grafana dashboard (seen in the image above) is here⁠. You can import it directly into your Grafana instance.

⁠Usage

Celery needs to be configured to send events to the broker which the exporter will collect. You can either enable this via Celery configuration or via the Celery CLI.

⁠Enable events using the CLI

To enable events in the CLI run the below command. Note that by default it doesn't send the task-sent event which needs to be configured⁠ in the configuration. The other events work out of the box.

$ celery -A <myproject> control enable_events

Enable events using the configuration:

# In celeryconfig.py
worker_send_task_events = True
task_send_sent_event = True

Configuration in Django:

# In settings.py
CELERY_WORKER_SEND_TASK_EVENTS = True
CELERY_TASK_SEND_SENT_EVENT = True

⁠Running the exporter

Using Docker:

docker run -p 9808:9808 danihodovic/celery-exporter --broker-url=redis://redis.service.consul/1

Using the Python binary (for-non Docker environments):

curl -L https://github.com/danihodovic/celery-exporter/releases/download/latest/celery-exporter -o ./celery-exporter
chmod+x ./celery-exporter
./celery-exporter --broker-url=redis://redis.service.consul/1

⁠Specifying optional broker transport options

While the default options may be fine for most cases, there may be a need to specify optional broker transport options. This can be done by specifying one or more --broker-transport-option parameters as follows:

docker run -p 9808:9808 danihodovic/celery-exporter --broker-url=redis://redis.service.consul/1 \
  --broker-transport-option global_keyprefix=danihodovic \
  --broker-transport-option visibility_timeout=7200

The list of available broker transport options can be found here: https://docs.celeryq.dev/projects/kombu/en/stable/reference/kombu.transport.redis.html⁠

⁠Specifying an optional retry interval

By default, celery-exporter will raise an exception and exit if there are any errors communicating with the broker. If preferred, one can have the celery-exporter retry connecting to the broker after a certain period of time in seconds via the --retry-interval parameter as follows:

docker run -p 9808:9808 danihodovic/celery-exporter --broker-url=redis://redis.service.consul/1 \
  --retry-interval=5

⁠Grafana Dashboards & Prometheus Alerts

Head over to the Celery-mixin in this subdirectory⁠ to generate rules and dashboards suited to your Prometheus setup.

⁠Metrics

Used in production at https://findwork.dev⁠.