llm/docs/runbooks/deploy-llama-cpp-gguf.md

Deploy llama.cpp GGUF

Цель: поднять GGUF-модель Devstral Small 2 24B Instruct 2512 Q4_K_M через llama-server на docker-gpu.cin.su.

Model

• Repo: bartowski/mistralai_Devstral-Small-2-24B-Instruct-2512-GGUF
• File: mistralai_Devstral-Small-2-24B-Instruct-2512-Q4_K_M.gguf
• Size: 14334438272 bytes
• Registry card: registry/model-cards/devstral-small-2-24b-instruct-2512-q4_k_m.yaml

Download

Resume-safe local download:

powershell -NoProfile -ExecutionPolicy Bypass -File scripts/download_devstral_gguf.ps1

The script can be re-run after network failures.

Compose

core/deploy/docker-gpu/llama-cpp/compose.yaml
core/deploy/docker-gpu/llama-cpp/.env.example

Run

powershell -NoProfile -ExecutionPolicy Bypass -File scripts/deploy_llama_cpp.ps1 -Pull

Проверить compose без запуска:

powershell -NoProfile -ExecutionPolicy Bypass -File scripts/deploy_llama_cpp.ps1 -ConfigOnly

Остановить сервис:

powershell -NoProfile -ExecutionPolicy Bypass -File scripts/deploy_llama_cpp.ps1 -Down

Check

python scripts/check_inference_endpoint.py --base-url http://docker-gpu.cin.su:8080 --expected-model devstral-1c-q4 --print

python scripts/smoke_chat.py --base-url http://docker-gpu.cin.su:8080 --model devstral-1c-q4