cbinckly/speaches repository overview

⁠Speaches for Jetson Orin

GPU-accelerated speech-to-text on NVIDIA Jetson devices. Provides an OpenAI-compatible API using faster-whisper.

⁠Quick Start

docker run -d \
  --name speaches \
  --runtime nvidia \
  -p 8000:8000 \
  -v speaches-models:/home/ubuntu/.cache/huggingface/hub \
  cbinckly/speaches:0.9.0-l4t-cuda-12.6.11-arch87

⁠Test

Test a transcription.

curl -X POST http://localhost:8000/v1/audio/transcriptions \
  -H "Content-Type: multipart/form-data" \
  -F "[email protected]" \
  -F "model=Systran/faster-whisper-small"

⁠Requirements

• Jetson Orin Nano, Orin NX, or AGX Orin (SM87 architecture)
• JetPack 6.x (L4T r36.4)
• NVIDIA Container Runtime (--runtime nvidia)

⁠Performance

~3 seconds to transcribe 20 seconds of audio using faster-whisper-small on Orin Nano.

⁠What's Different

This image compiles ctranslate2 from source with CUDA and cuDNN support. The standard PyPI wheels for aarch64 are CPU-only and will not use the GPU.

Key build changes from upstream:

L4T base images (required for Jetson) cuDNN from Jetson apt repository OpenBLAS instead of Intel MKL CUDA architecture set to SM87

⁠Models

Models download automatically on first use. Mount a volume to persist them:

-v speaches-models:/home/ubuntu/.cache/huggingface/hub

Recommended models:

• Systran/faster-whisper-small (~1GB, good balance)
• Systran/faster-whisper-medium (~3GB, higher accuracy)

⁠Links

Speaches upstream⁠ Build documentation / Medium article

⁠License

MIT (follows upstream speaches license)