osodevops/rabbitmq-backup

•Updated about 2 months ago

Image

1.2K

osodevops/rabbitmq-backup repository overview

⁠rabbitmq-backup

Non-destructive RabbitMQ backup and restore with point-in-time recovery

rabbitmq-backup is a production-grade tool written in Rust for backing up and restoring RabbitMQ queues, streams, and definitions to cloud storage or a local filesystem. It is designed for disaster recovery workflows where the broker must keep running and source queues must not be drained.

It supports classic queues, quorum queues, RabbitMQ Streams, topology backup and restore via the Management API, point-in-time restore filters, resumable backup/restore checkpoints, and S3-compatible object storage.

⁠Features

Non-destructive queue backup - backs up messages without acking or consuming them from classic and quorum queues
RabbitMQ Streams support - uses the native Stream Protocol for offset-based stream backups
Definitions backup and restore - exports and imports vhosts, queues, exchanges, bindings, users, permissions, policies, and parameters
Selective definitions restore - restores only selected vhosts, queues, or exchanges from a full definitions backup
Point-in-time restore - filters restored messages by backup or original message timestamp
Resumable backups - SQLite checkpoints track completed queues, stream offsets, and reusable segment files
Resumable restores - optional restore checkpoint database skips completed queues on retry
Rollback safety - writes a target-cluster definitions snapshot before importing backed-up definitions
Strict publish failure handling - mandatory publishing and publisher confirms fail the restore on missing routes or broker nacks
Missing queue creation - can declare missing target queues for direct-to-queue restores
Multi-vhost restore - queue, vhost, and exchange remapping for migration and DR workflows
Multi-cloud storage - S3, Azure Blob Storage, Google Cloud Storage, and local filesystem via object_store
Compression and integrity - zstd/lz4 segment compression with checksum validation
Deployment agnostic - single binary for bare metal, VMs, Docker, and Kubernetes
Observable - Prometheus metrics endpoint for backup monitoring

⁠Installation

Download the latest binary from the GitHub Releases⁠ page.

⁠macOS / Linux (Homebrew)

brew install osodevops/tap/rabbitmq-backup

⁠Linux / macOS (Shell Installer)

curl --proto '=https' --tlsv1.2 -LsSf https://github.com/osodevops/rabbitmq-backup/releases/latest/download/rabbitmq-backup-cli-installer.sh | sh

⁠Linux (Manual)

Download the appropriate binary for your architecture from releases⁠:

# Example for x86_64
curl -LO https://github.com/osodevops/rabbitmq-backup/releases/latest/download/rabbitmq-backup-cli-x86_64-unknown-linux-gnu.tar.xz
tar -xJf rabbitmq-backup-cli-x86_64-unknown-linux-gnu.tar.xz
sudo mv rabbitmq-backup /usr/local/bin/

⁠Windows (PowerShell Installer)

powershell -ExecutionPolicy ByPass -c "irm https://github.com/osodevops/rabbitmq-backup/releases/latest/download/rabbitmq-backup-cli-installer.ps1 | iex"

⁠Windows (Scoop)

scoop bucket add oso https://github.com/osodevops/scoop-bucket.git
scoop install rabbitmq-backup

⁠Docker

docker pull osodevops/rabbitmq-backup
docker run --rm -v /path/to/config:/config osodevops/rabbitmq-backup \
  backup --config /config/example-backup.yaml

⁠From Source

git clone https://github.com/osodevops/rabbitmq-backup.git
cd rabbitmq-backup
cargo build --release

Binary location:

./target/release/rabbitmq-backup

⁠Docker From Source

Build the image locally:

docker build -t rabbitmq-backup:test .

Run the CLI:

docker run --rm -v "$PWD/config:/config" rabbitmq-backup:test \
  backup --config /config/example-backup.yaml

⁠Try It Locally

Start RabbitMQ and MinIO:

docker compose up -d rabbitmq minio

Run the local verification bundle:

./scripts/verify-local.sh quick
./scripts/verify-local.sh integration
./scripts/verify-local.sh s3

⁠Quick Start

⁠Backup

Create backup.yaml:

mode: backup
backup_id: "daily-backup-001"

source:
  amqp_url: "amqp://guest:guest@localhost:5672/%2f"
  management_url: "http://localhost:15672"
  management_username: guest
  management_password: guest
  queues:
    include: ["orders-*", "payments-*"]
    exclude: ["*-dlq"]

storage:
  backend: filesystem
  path: ./backups

backup:
  compression: zstd
  include_definitions: true
  stop_at_current_depth: true

Run the backup:

rabbitmq-backup backup --config backup.yaml

⁠Restore

Create restore.yaml:

mode: restore
backup_id: "daily-backup-001"

target:
  amqp_url: "amqp://guest:guest@localhost:5672/%2f"
  management_url: "http://localhost:15672"
  management_username: guest
  management_password: guest

storage:
  backend: filesystem
  path: ./backups

restore:
  restore_definitions: true
  publish_mode: exchange
  publisher_confirms: true
  checkpoint_state: ./restore-checkpoint.db

Run the restore:

rabbitmq-backup restore --config restore.yaml

⁠Message-Only Restore Into Missing Queues

restore:
  restore_definitions: false
  publish_mode: direct-to-queue
  create_missing_queues: true
  publisher_confirms: true

create_missing_queues requires target.management_url, target.management_username, and target.management_password.

⁠Why OSO RabbitMQ Backup?

Capability	OSO RabbitMQ Backup	RabbitMQ definitions export	Filesystem snapshot	Shovel/Federation
Message backup	Yes	No	Yes	Replication only
Non-destructive queue reads	Yes	N/A	Requires broker/filesystem coordination	No
Broker stays online	Yes	Yes	Not always	Yes
Point-in-time restore	Yes	No	Snapshot only	No
Cloud object storage	S3, Azure, GCS	Manual	Manual	No
Definitions backup	Yes	Yes	Yes	No
Selective definitions restore	Yes	Manual	No	No
Stream queue support	Yes	Topology only	Snapshot only	Limited
Resumable backup/restore	Yes	No	Snapshot dependent	No
License	MIT	RabbitMQ toolchain	Infrastructure dependent	RabbitMQ toolchain

OSO RabbitMQ Backup is for teams that need cold, air-gapped RabbitMQ backups without stopping the broker and without turning a backup job into a destructive consumer.

⁠When Not To Use RabbitMQ Backup

Real-time replication - use RabbitMQ Federation, Shovel, streams replication, or your platform replication tooling.
Broker filesystem disaster recovery - if you need exact broker node state, plan broker-level snapshots with RabbitMQ operational guidance.
Application-level replay transformation - rabbitmq-backup preserves message payloads/properties and does not transform schemas or business events.

⁠Documentation

Full documentation is available at rabbitmqbackup.com⁠.

Document	Description
Quickstart⁠	Get started in minutes
Configuration Reference⁠	Full YAML configuration schema
CLI Reference⁠	Commands, flags, and outputs
Storage Format⁠	Manifest, segment, definitions, and checkpoint layout
Restore Safety⁠	Rollback snapshots, checkpoints, selective definitions, and publish failures
Stream Backup⁠	RabbitMQ Streams backup and restore
Backup to S3⁠	S3 and S3-compatible storage setup

⁠CLI Reference

# Backup and restore
rabbitmq-backup backup --config backup.yaml
rabbitmq-backup restore --config restore.yaml

# Inspect backups
rabbitmq-backup list --path ./backups
rabbitmq-backup list --path s3://bucket/prefix
rabbitmq-backup describe --path ./backups --backup-id backup-001 --format json

# Validate backup integrity
rabbitmq-backup validate --path ./backups --backup-id backup-001
rabbitmq-backup validate --path ./backups --backup-id backup-001 --deep

# Definitions-only workflows
rabbitmq-backup definitions-export --config config/example-backup.yaml --output defs.json
rabbitmq-backup definitions-import --config config/example-restore.yaml --input defs.json

# Shell completions
rabbitmq-backup completions bash > ~/.bash_completion.d/rabbitmq-backup
rabbitmq-backup completions zsh > ~/.zfunc/_rabbitmq-backup

⁠Storage Layout

Backups are stored in a structured format:

{prefix}/{backup_id}/
|-- manifest.json
|-- definitions/
|   |-- definitions.json.zst
|   `-- rollback-before-import-<timestamp>.zst
|-- state/
|   `-- offsets.db
`-- queues/
    `-- {vhost}/
        `-- {queue_name}/
            |-- segment-0001.zst
            `-- segment-0002.zst

The default vhost / is encoded as _default in storage paths.

⁠Metrics and Monitoring

Enable Prometheus metrics in the config:

metrics:
  enabled: true
  bind_address: "0.0.0.0"
  port: 8080

The metrics endpoint is available at /metrics.

⁠Performance

Performance depends on broker load, queue type, message size, storage latency, and compression settings. Useful tuning levers:

Setting	Use
`backup.max_concurrent_queues`	Increase parallel queue backup work
`backup.prefetch_count`	Tune AMQP read batching
`backup.segment_max_bytes`	Control segment size and object count
`backup.compression`	Choose `zstd`, `lz4`, or `none`
`restore.produce_batch_size`	Tune restore publish batching
`restore.rate_limit_messages_per_sec`	Protect target brokers during restore

⁠Building from Source

Requirements:

Rust stable
Docker, for integration tests

cargo build --release
cargo test --workspace
RUST_LOG=debug cargo run -p rabbitmq-backup-cli -- --help

⁠Running Tests

# Format, clippy, unit tests, docs, and release build
./scripts/verify-local.sh quick

# Live RabbitMQ integration tests
./scripts/verify-local.sh integration

# S3-compatible MinIO end-to-end test
./scripts/verify-local.sh s3

# All local verification modes
./scripts/verify-local.sh all

For a clean test pass on another machine, see docs/test-handoff.md⁠.

⁠Project Structure

rabbitmq-backup/
|-- crates/
|   |-- rabbitmq-backup-core/
|   |   |-- src/
|   |   |   |-- amqp/          # AMQP 0-9-1 client and TLS support
|   |   |   |-- backup/        # Backup engine, queue reader, stream reader
|   |   |   |-- definitions/   # Management API definitions export/import
|   |   |   |-- offset_store/  # SQLite checkpoint state
|   |   |   |-- restore/       # Restore engine and publisher
|   |   |   |-- storage/       # S3, Azure, GCS, filesystem
|   |   |   |-- stream/        # RabbitMQ Stream Protocol client
|   |   |   `-- metrics/       # Prometheus metrics
|   |   `-- tests/             # Integration tests
|   `-- rabbitmq-backup-cli/   # CLI binary
|-- config/                    # Example configs
|-- scripts/                   # Local verification scripts
`-- docs/                      # Test handoff and local notes

osodevops/kafka-backup⁠ - sister project for Apache Kafka backup

⁠Looking for Enterprise RabbitMQ Support?

OSO⁠ engineers help teams deploy, operate, recover, and maintain production messaging platforms.

Need help planning RabbitMQ disaster recovery, validating backups, or recovering a degraded cluster? Contact OSO⁠ or email [email protected].

⁠Contributing

Contributions are welcome:

Report bugs with a minimal reproduction and relevant config snippets.
Request features through GitHub Issues.
Improve docs and examples in the docs repository: osodevops/rabbitmq-backup-docs⁠.
Run ./scripts/verify-local.sh quick before opening a pull request.

⁠License

OSO RabbitMQ Backup is licensed under the MIT License⁠.

⁠Acknowledgments

Built with these Rust crates and projects:

amq-protocol⁠ - AMQP 0-9-1 protocol implementation
rabbitmq-stream-client⁠ - RabbitMQ Stream Protocol client
object_store⁠ - cloud storage abstraction
tokio⁠ - async runtime
zstd⁠ and lz4_flex⁠ - compression

Made by OSO⁠.

Tag summary

Recent tags

Content type

Image

Digest

sha256:13331ac1d…

Size

38.1 MB

Last updated

about 2 months ago

docker pull osodevops/rabbitmq-backup