osodevops/kafka-partition-remapper repository overview

⁠Kafka Partition Remapping Proxy

A high-performance Kafka proxy that virtualizes partition counts, enabling cost reduction and seamless partition management

⁠ ⁠ ⁠

A lightweight, developer-friendly proxy for Apache Kafka that transparently remaps partitions between clients and brokers. Designed to make partition migrations, traffic rebalancing, and topology changes simple, safe, and zero-touch for your producers and consumers.

⁠Why Kafka Partition Remapping Proxy?

Running Kafka at scale often means dealing with partition constraints:

• Cost pressure: Managed Kafka services like Confluent Cloud charge per partition. High-cardinality use cases (multi-tenant SaaS, IoT, event sourcing) need thousands of logical partitions but this becomes prohibitively expensive.
• Migration complexity: Changing partition layouts requires complex coordination, custom tooling, or application changes.
• Operational overhead: Rebalancing hot partitions or evolving topology disrupts clients.

Kafka Partition Remapping Proxy solves this by sitting between your clients and your Kafka cluster, handling all partition remapping logic transparently. Producers and consumers connect to the proxy as if it were Kafka, while the proxy takes care of routing and remapping behind the scenes.

Client sees:     1000 virtual partitions  ($$$$$)
Kafka has:       100 physical partitions  ($)
Compression:     10:1 cost reduction

⁠Key Benefits

⁠Zero Code Changes

Keep your client config and application code untouched. Point them at the proxy and let it handle partition remapping and routing logic.

⁠Simple, Declarative Configuration

Define your partition remapping rules with a clean YAML configuration. No scripting or operational gymnastics required.

⁠Safe Migrations and Rebalancing

Move partitions, rebalance load, or change topic layouts with predictable behavior and minimal risk of client disruptions.

⁠Transparent to Clients

The proxy maintains Kafka semantics, so your existing tooling, metrics, and consumers continue to work as expected.

⁠High Performance

Written in async Rust with Tokio for minimal latency overhead (≤1-3ms p99) and high throughput.

⁠Optimized for Operations

Built to be easy to deploy, observe, and operate in modern Kubernetes and cloud environments. Prometheus metrics built-in.

⁠Use Cases

⁠Reduce Managed Kafka Costs

Use 1000 virtual partitions while paying for only 100 physical partitions. Achieve 10:1 cost reduction on partition-based pricing.

⁠Partition Migration Without Downtime

Move partitions between brokers or clusters without coordinating rolling client restarts or application releases.

⁠Load Rebalancing for Hot Partitions

Smooth out hot spots by transparently remapping partitions, while clients continue to write/read using the same partition IDs.

⁠Progressive Topology Changes

Evolve your Kafka topology over time with a proxy layer that decouples client expectations from physical partition layouts.

⁠Multi-Tenant Isolation

Give each tenant their own virtual partition space while efficiently sharing physical resources.

⁠Why Kafka Partition Remapper?

Why choose Partition Remapper? Open source, lightweight, and purpose-built for partition cost optimization with minimal overhead. No enterprise license required.

⁠How It Works

┌─────────────┐     ┌─────────────────────┐     ┌─────────────┐
│   Client    │────▶│  Partition Proxy    │────▶│    Kafka    │
│ (v.part 42) │     │  42 → phys.part 2   │     │  (10 parts) │
└─────────────┘     └─────────────────────┘     └─────────────┘

1. Clients connect to the proxy instead of directly to the Kafka cluster
2. The proxy intercepts metadata and data traffic
3. Based on configuration, it remaps logical partitions to physical partitions
4. To the client, everything still looks like a regular Kafka cluster

Partition Mapping:

physical_partition = virtual_partition % physical_partitions

Offset Translation:

physical_offset = (virtual_group * offset_range) + virtual_offset

Each virtual partition group gets its own offset space (default: 2^40 offsets), ensuring complete isolation.

⁠Quick Start

⁠1. Create Configuration

listen:
  address: "0.0.0.0:9092"

kafka:
  bootstrap_servers:
    - "your-kafka:9092"

mapping:
  virtual_partitions: 100
  physical_partitions: 10

metrics:
  enabled: true
  address: "0.0.0.0:9090"

⁠2. Start the Proxy

# Using Docker
docker run --rm \
  -v /path/to/config.yaml:/config/config.yaml \
  osodevops/kafka-partition-remapper --config /config/config.yaml

# Or from source
cargo build --release
./target/release/kafka-partition-proxy --config config.yaml

⁠3. Connect Clients

Point your Kafka clients to the proxy instead of Kafka:

# Before (direct to Kafka)
kafka-console-producer --bootstrap-server kafka:9092 --topic my-topic

# After (through proxy)
kafka-console-producer --bootstrap-server proxy:9092 --topic my-topic

⁠Docker Compose

cd docker
docker compose up -d

See the Quickstart Guide⁠ for detailed instructions.

⁠Try It Yourself

Get hands-on with example configurations in the config/⁠ directory:

# Run with an example config
docker run -v $(pwd)/config:/config osodevops/kafka-partition-remapper \
  --config /config/example.yaml

⁠Installation

⁠macOS (Homebrew)

brew install osodevops/tap/kafka-partition-proxy

⁠Linux/macOS (Shell Installer)

curl --proto '=https' --tlsv1.2 -LsSf https://github.com/osodevops/kafka-partition-remapper/releases/latest/download/kafka-remapper-cli-installer.sh | sh

⁠Windows (PowerShell Installer)

powershell -ExecutionPolicy ByPass -c "irm https://github.com/osodevops/kafka-partition-remapper/releases/latest/download/kafka-remapper-cli-installer.ps1 | iex"

⁠Windows (Scoop)

scoop bucket add oso https://github.com/osodevops/scoop-bucket.git
scoop install kafka-partition-proxy

⁠Docker

docker pull osodevops/kafka-partition-remapper
docker run --rm osodevops/kafka-partition-remapper --help

⁠From Source

cargo install --git https://github.com/osodevops/kafka-partition-remapper kafka-remapper-cli

⁠Configuration

⁠Features

• Transparent Kafka proxy for producers and consumers
• Partition-level remapping with declarative configuration
• Offset translation maintaining consumer group semantics
• High performance async Rust with Tokio
• Observable with Prometheus metrics
• Containerized for Kubernetes deployments
• Minimal dependencies and small operational footprint

⁠Metrics

⁠Supported Kafka APIs

⁠Performance

Benchmarks available via cargo bench.

⁠Documentation

⁠Limitations

• Global mapping by default — Same V:P ratio for all topics (per-topic overrides available)
• Kafka 3.6+ — Tested with Confluent Cloud and Apache Kafka 3.6+
• Fixed partition counts — Topics must have exactly physical_partitions partitions

⁠Project Structure

kafka-partition-remapper/
├── crates/
│   ├── kafka-remapper-core/     # Core library
│   │   ├── src/
│   │   │   ├── config.rs        # Configuration
│   │   │   ├── error.rs         # Error types
│   │   │   ├── remapper/        # Partition mapping logic
│   │   │   ├── network/         # TCP listener, codec
│   │   │   ├── broker/          # Broker connections
│   │   │   ├── handlers/        # Protocol handlers
│   │   │   ├── metrics/         # Prometheus metrics
│   │   │   └── testing/         # Test harness & mock broker
│   │   ├── tests/               # Integration tests
│   │   └── benches/             # Performance benchmarks
│   └── kafka-remapper-cli/      # CLI binary
├── config/                      # Example configs
├── docker/                      # Docker setup
└── docs/                        # Documentation

⁠Building

# Build
cargo build --release

# Test (131 tests)
cargo test --features testing

# Run benchmarks
cargo bench

# Run with debug logging
RUST_LOG=debug cargo run -p kafka-remapper-cli -- --config config.yaml

⁠Contributing

Contributions are welcome! Please read our contributing guidelines before submitting a PR.

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

⁠License

Kafka Partition Remapping Proxy is licensed under the Apache License 2.0⁠.

⁠Summary

Kafka Partition Remapping Proxy is an easy-to-deploy Kafka proxy that simplifies partition management, reduces costs on managed Kafka services, and enables seamless topology changes without requiring changes to your client applications. By centralizing partition remapping logic in a lightweight Rust proxy, it helps Kafka platform teams deliver safer, faster infrastructure changes while keeping the developer experience simple.

Made with Rust by OSO⁠