56 lines
1.4 KiB
Markdown
56 lines
1.4 KiB
Markdown
# Deploy llama.cpp GGUF
|
|
|
|
Цель: поднять GGUF-модель `Devstral Small 2 24B Instruct 2512 Q4_K_M` через `llama-server` на `docker-gpu.cin.su`.
|
|
|
|
## Model
|
|
|
|
- Repo: `bartowski/mistralai_Devstral-Small-2-24B-Instruct-2512-GGUF`
|
|
- File: `mistralai_Devstral-Small-2-24B-Instruct-2512-Q4_K_M.gguf`
|
|
- Size: `14334438272` bytes
|
|
- Registry card: `registry/model-cards/devstral-small-2-24b-instruct-2512-q4_k_m.yaml`
|
|
|
|
## Download
|
|
|
|
Resume-safe local download:
|
|
|
|
```powershell
|
|
powershell -NoProfile -ExecutionPolicy Bypass -File scripts/download_devstral_gguf.ps1
|
|
```
|
|
|
|
The script can be re-run after network failures.
|
|
|
|
## Compose
|
|
|
|
```text
|
|
core/deploy/docker-gpu/llama-cpp/compose.yaml
|
|
core/deploy/docker-gpu/llama-cpp/.env.example
|
|
```
|
|
|
|
## Run
|
|
|
|
```powershell
|
|
powershell -NoProfile -ExecutionPolicy Bypass -File scripts/deploy_llama_cpp.ps1 -Pull
|
|
```
|
|
|
|
Проверить compose без запуска:
|
|
|
|
```powershell
|
|
powershell -NoProfile -ExecutionPolicy Bypass -File scripts/deploy_llama_cpp.ps1 -ConfigOnly
|
|
```
|
|
|
|
Остановить сервис:
|
|
|
|
```powershell
|
|
powershell -NoProfile -ExecutionPolicy Bypass -File scripts/deploy_llama_cpp.ps1 -Down
|
|
```
|
|
|
|
## Check
|
|
|
|
```powershell
|
|
python scripts/check_inference_endpoint.py --base-url http://docker-gpu.cin.su:8080 --expected-model devstral-1c-q4 --print
|
|
```
|
|
|
|
```powershell
|
|
python scripts/smoke_chat.py --base-url http://docker-gpu.cin.su:8080 --model devstral-1c-q4
|
|
```
|