yodsarun/kilovolt-proxy repository overview

⁠Kilovolt (kvlt) ⚡ — The Bankruptcy Shield for AI Stream Engines

Kilovolt (kvlt) is a hyper-optimized, high-throughput asynchronous reverse proxy gateway written in Rust. It acts as an active financial circuit breaker and Bankruptcy Shield for independent developers, startups, and autonomous coding agents running AI integrations on low-resource hardware.

🚀 Want a hosted version? Join the Kilovolt Cloud Waitlist.

⁠📊 Rust vs. Go vs. Python Performance Comparison

Kilovolt is engineered in Rust to target resource-constrained $5/month virtual private servers (VPS) where every megabyte of RAM and CPU cycle counts.

⁠🚀 Key Features

• Zero-Copy Stream Piping: Pipes byte-streams asynchronously with $O(1)$ space complexity, keeping memory footprint flat regardless of the size or duration of chat completions.
• Mid-Stream Circuit Breaker: Actively decodes and tokenizes incoming SSE chunks on the fly. The exact millisecond cumulative spend crosses your default budget, the TCP socket is severed.
• Pre-Flight Prompt Validation: Audits prompt token counts upfront using local BPE tokenization, projecting costs and rejecting requests with 429 errors before querying upstream APIs.
• Multi-Tier Token Budgets: Enforces per-step prompt size limits, per-pipeline stuck-loop prevention, and per-day token ceiling caps (which reset automatically at server-midnight).
• Connection Abortion: Monitors client sockets. If a developer aborts a query or closes a tab, the proxy instantly drops the upstream socket to prevent "ghost token" billing.
• Google Gemini Translation Layer: Maps standard OpenAI-compatible completions payloads directly to Gemini, converting native camelCase output formats back to OpenAI choices deltas on the fly.

⁠📦 Container Deployment (Under 30 Seconds)

⁠1. Setup Environment

Create a .env file containing your configurations:

KILOVOLT_PORT=8080
KILOVOLT_DEFAULT_BUDGET=5.00
RUST_LOG=info

# Optional: Configure Token Budgeting Limits
KILOVOLT_PER_STEP_TOKENS=2048
KILOVOLT_PER_PIPELINE_TOKENS=10000
KILOVOLT_PER_DAY_TOKENS=100000

⁠2. Run the Gateway Container

Expose the proxy port and pass the environment variables file:

docker run -d \
  --name kilovolt-proxy \
  --env-file .env \
  -p 8080:8080 \
  yodsarun/kilovolt-proxy:1.3.1

⁠3. Or deploy via Docker Compose

Mount settings inside a docker-compose.yml config:

version: '3.8'
services:
  kilovolt-proxy:
    image: yodsarun/kilovolt-proxy:1.3.1
    container_name: kilovolt-proxy
    ports:
      - "8080:8080"
    env_file: .env
    deploy:
      resources:
        limits:
          memory: 30M
    restart: unless-stopped

⁠🔌 API Client Integration (Drop-in Replacement)

Redirect your OpenAI SDK to Kilovolt and track spend ledger using the X-User-ID header.

Python SDK Integration

import os
from openai import OpenAI
client = OpenAI(
    api_key=os.environ.get("OPENAI_API_KEY", "your-api-key"),
    base_url="<http://127.0.0.1:8080/v1>" # Target local Kilovolt instance
)
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Explain zero-copy streams."}],
    stream=True,
    extra_headers={
        "X-User-ID": "developer_alice",
        "X-Pipeline-ID": "pipeline_run_456",
        "X-Pipeline-Name": "DocumentSummarization",
        "X-Step-Name": "BPETextChunking"
    }
)
for chunk in response:
    content = chunk.choices[0].delta.content
    if content:
        print(content, end="", flush=True)

📊 Analytics Dashboard Kilovolt exposes a built-in admin dashboard. Navigate to http://localhost:8080/dashboard⁠ in your browser to inspect memory usage, average latencies, active agents, and rolling transaction costs.