Initial SQL-only 1C adapter baseline
This commit is contained in:
@@ -0,0 +1,55 @@
|
||||
# Deploy llama.cpp GGUF
|
||||
|
||||
Цель: поднять GGUF-модель `Devstral Small 2 24B Instruct 2512 Q4_K_M` через `llama-server` на `docker-gpu.cin.su`.
|
||||
|
||||
## Model
|
||||
|
||||
- Repo: `bartowski/mistralai_Devstral-Small-2-24B-Instruct-2512-GGUF`
|
||||
- File: `mistralai_Devstral-Small-2-24B-Instruct-2512-Q4_K_M.gguf`
|
||||
- Size: `14334438272` bytes
|
||||
- Registry card: `registry/model-cards/devstral-small-2-24b-instruct-2512-q4_k_m.yaml`
|
||||
|
||||
## Download
|
||||
|
||||
Resume-safe local download:
|
||||
|
||||
```powershell
|
||||
powershell -NoProfile -ExecutionPolicy Bypass -File scripts/download_devstral_gguf.ps1
|
||||
```
|
||||
|
||||
The script can be re-run after network failures.
|
||||
|
||||
## Compose
|
||||
|
||||
```text
|
||||
core/deploy/docker-gpu/llama-cpp/compose.yaml
|
||||
core/deploy/docker-gpu/llama-cpp/.env.example
|
||||
```
|
||||
|
||||
## Run
|
||||
|
||||
```powershell
|
||||
powershell -NoProfile -ExecutionPolicy Bypass -File scripts/deploy_llama_cpp.ps1 -Pull
|
||||
```
|
||||
|
||||
Проверить compose без запуска:
|
||||
|
||||
```powershell
|
||||
powershell -NoProfile -ExecutionPolicy Bypass -File scripts/deploy_llama_cpp.ps1 -ConfigOnly
|
||||
```
|
||||
|
||||
Остановить сервис:
|
||||
|
||||
```powershell
|
||||
powershell -NoProfile -ExecutionPolicy Bypass -File scripts/deploy_llama_cpp.ps1 -Down
|
||||
```
|
||||
|
||||
## Check
|
||||
|
||||
```powershell
|
||||
python scripts/check_inference_endpoint.py --base-url http://docker-gpu.cin.su:8080 --expected-model devstral-1c-q4 --print
|
||||
```
|
||||
|
||||
```powershell
|
||||
python scripts/smoke_chat.py --base-url http://docker-gpu.cin.su:8080 --model devstral-1c-q4
|
||||
```
|
||||
Reference in New Issue
Block a user