An OpenAI-compatible LLM chat and embedding server: https://github.com/LlamaEdge/LlamaEdge
3.6K
This image starts an OpenAI-compatible API server. It contains
How to use it: https://github.com/LlamaEdge/LlamaEdge/tree/main/docker
On Mac, or Linux / Windows with CPU only:
docker run --name api-server \
-p 8080:8080 \
secondstate/qwen-2-0.5b-allminilm-2:latest
On Linux / Windows with Nvidia GPU CUDA 11
docker run --name api-server \
-p 8080:8080 --device nvidia.com/gpu=all \
secondstate/qwen-2-0.5b-allminilm-2:cuda11
On Linux / Windows with Nvidia GPU CUDA 12
docker run --name api-server \
-p 8080:8080 --device nvidia.com/gpu=all \
secondstate/qwen-2-0.5b-allminilm-2:cuda12
Content type
Image
Digest
sha256:1bb7a161b…
Size
519.7 MB
Last updated
over 2 years ago
docker pull secondstate/qwen-2-0.5b-allminilm-2