# Deploy llama.cpp GGUF

Цель: поднять GGUF-модель `Devstral Small 2 24B Instruct 2512 Q4_K_M` через `llama-server` на `docker-gpu.cin.su`.

## Model

- Repo: `bartowski/mistralai_Devstral-Small-2-24B-Instruct-2512-GGUF`
- File: `mistralai_Devstral-Small-2-24B-Instruct-2512-Q4_K_M.gguf`
- Size: `14334438272` bytes
- Registry card: `registry/model-cards/devstral-small-2-24b-instruct-2512-q4_k_m.yaml`

## Download

Resume-safe local download:

```powershell
powershell -NoProfile -ExecutionPolicy Bypass -File scripts/download_devstral_gguf.ps1
```

The script can be re-run after network failures.

## Compose

```text
core/deploy/docker-gpu/llama-cpp/compose.yaml
core/deploy/docker-gpu/llama-cpp/.env.example
```

## Run

```powershell
powershell -NoProfile -ExecutionPolicy Bypass -File scripts/deploy_llama_cpp.ps1 -Pull
```

Проверить compose без запуска:

```powershell
powershell -NoProfile -ExecutionPolicy Bypass -File scripts/deploy_llama_cpp.ps1 -ConfigOnly
```

Остановить сервис:

```powershell
powershell -NoProfile -ExecutionPolicy Bypass -File scripts/deploy_llama_cpp.ps1 -Down
```

## Check

```powershell
python scripts/check_inference_endpoint.py --base-url http://docker-gpu.cin.su:8080 --expected-model devstral-1c-q4 --print
```

```powershell
python scripts/smoke_chat.py --base-url http://docker-gpu.cin.su:8080 --model devstral-1c-q4
```