lordraw/llmproxy repository overview

⁠llmproxy

One proxy, three APIs, many providers. llmproxy speaks the Ollama, OpenAI and llama.cpp HTTP protocols at the same time and relays every request to the provider that owns the requested model — any OpenAI-compatible endpoint (NVIDIA by default, plus OpenAI, Mistral, Groq, OpenRouter, vLLM, a local Ollama…), Azure OpenAI, Anthropic and Google Gemini, the last two translated natively to and from the OpenAI shape. To your tools it looks like a local LLM server; the inference runs wherever you configured it.

Point Open WebUI, an IDE plugin, a chat frontend or an agent framework at llmproxy — no client change beyond the base URL — and serve responses from any hosted model. With no configuration file at all, a single NVIDIA provider is built from the NVIDIA_* variables.

• 🐙 Source & docs: https://github.com/lordraw77/llmproxy⁠
• 🐳 Image: lordraw/llmproxy

⁠Features

• Triple API surface — Ollama (/api/chat, /api/generate, /api/tags, /api/show, /api/version), OpenAI (/v1/chat/completions, /v1/completions, /v1/models, /v1/models/<id>, /v1/embeddings) and llama.cpp (/completion, /props).
• Embeddings — OpenAI /v1/embeddings and Ollama /api/embed + legacy /api/embeddings, with a dedicated default embeddings model.
• Streaming — SSE for OpenAI/llama.cpp, NDJSON for Ollama; token usage is logged and re-exposed.
• Multi-model — advertise a whole list of models; clients with a model picker (e.g. Open WebUI) just work.
• Multi-provider — serve several upstreams at once via providers.toml (OpenAI-compatible, Azure, Anthropic, Gemini); all their models are exposed together as provider:model and routed to the owning provider. With no file, a single NVIDIA provider is built from the NVIDIA_* vars (zero-config).
• Vision — pass image_url content to vision-capable models.
• Resilient — configurable upstream timeout and automatic retry with exponential backoff on 429/5xx/network errors (honours Retry-After).
• Response caching — optional per-worker in-memory cache for non-streaming completions and embeddings, with configurable TTL and size (CACHE_ENABLED/CACHE_TTL/CACHE_MAX_SIZE/CACHE_POLICY) and hit/miss stats at /stats.
• Optional inbound auth — protect the proxy with PROXY_API_KEY (Authorization: Bearer or X-Api-Key); / and /health stay open for health-checks.
• Audit trail — optional (AUDIT_ENABLED): one structured record per request — prompt, completion, provider and native model, sampling parameters, token usage, retries, latency and time-to-first-token, and the conversation it belongs to. Written to a rotating file by a background thread, so the request never waits for it; under back-pressure records are dropped, never delayed.
• Observability — per-request correlation IDs, structured logging, /health with an optional live upstream probe (?upstream=1), plus a live /stats dashboard (and /stats.json) with request/latency/token metrics, cache and audit counters, and a process-manager view (PID, workers, memory, uptime).
• Clean architecture — a small layered Python/Flask package (config · domain · upstream · services · web), no database, run under gunicorn.

⁠Quick start

The zero-config path needs a valid NVIDIA API key (nvapi-…) from build.nvidia.com⁠. To serve several providers instead, mount a providers.toml and point PROVIDERS_CONFIG at it (-v ./providers.toml:/config/providers.toml:ro); see the configuration guide⁠.

⁠docker run

docker run -d --name llmproxy -p 11434:11434 \
  -e NVIDIA_API_KEY=nvapi-xxxxxxxx \
  -e NVIDIA_MODELS=meta/llama-3.1-8b-instruct,meta/llama-3.2-11b-vision-instruct \
  lordraw/llmproxy:latest

⁠docker compose

services:
  llmproxy:
    image: lordraw/llmproxy:latest
    container_name: llmproxy
    restart: unless-stopped
    env_file: .env
    ports:
      - "11434:11434"
    volumes:
      # Only needed with AUDIT_ENABLED: keeps the audit trail on the host.
      - ./logs:/app/logs

⁠Smoke test

# Discovery
curl http://localhost:11434/v1/models

# OpenAI chat
curl http://localhost:11434/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"meta/llama-3.1-8b-instruct",
       "messages":[{"role":"user","content":"Say hello."}]}'

# Ollama chat
curl http://localhost:11434/api/chat \
  -d '{"model":"meta/llama-3.1-8b-instruct","stream":false,
       "messages":[{"role":"user","content":"Say hello."}]}'

# Health (+ live upstream check)
curl "http://localhost:11434/health?upstream=1"

Using it as an Ollama backend (e.g. Open WebUI): set the Ollama base URL to http://<host>:11434. As an OpenAI backend: base URL http://<host>:11434/v1.

⁠Configuration

All configuration is via environment variables.

Minimal .env:

NVIDIA_API_KEY=nvapi-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
NVIDIA_MODELS=meta/llama-3.1-8b-instruct,meta/llama-3.2-11b-vision-instruct
LOG_TZ=Europe/Rome
# PROXY_API_KEY=change-me   # uncomment to require inbound auth
# AUDIT_ENABLED=on          # uncomment to record every request (needs ./logs mounted)

⁠Image tags

The image runs as a non-root user, exposes port 11434, includes a Docker HEALTHCHECK, and is served by gunicorn with threaded workers (compatible with SSE streaming). It is built for linux/amd64 (extendable to other platforms via the Makefile's PLATFORMS).

⁠Health & monitoring

• GET /health — liveness plus basic config (api_key_configured, number of models, default model).
• GET /health?upstream=1 — also probes every configured provider; returns status: degraded (HTTP 503) if one is unreachable.
• GET /stats — self-contained, auto-refreshing HTML dashboard with live statistics, metrics (requests, latency, tokens, upstream calls, cache and audit counters) and the process-manager view (PID, worker pool, memory, uptime). Open it in a browser at http://<host>:11434/stats.
• GET /stats.json — the same data as JSON for scraping/scripting.

⁠Smoke test (metrics)

curl http://localhost:11434/stats.json      # JSON snapshot
# open http://localhost:11434/stats in a browser for the dashboard

⁠Links

• GitHub (source, full docs, issues): https://github.com/lordraw77/llmproxy⁠
• Configuration reference: https://github.com/lordraw77/llmproxy/blob/main/docs/configuration.md⁠
• Audit trail: https://github.com/lordraw77/llmproxy/blob/main/docs/audit.md⁠
• NVIDIA API keys & models: https://build.nvidia.com/⁠

⁠License

Released under the MIT License — free to use, copy, modify, and distribute, with attribution and no warranty. See the LICENSE⁠ in the GitHub repository for the full text.