389 lines
15 KiB
Markdown
389 lines
15 KiB
Markdown
# Model Chat UI Service
|
|
|
|
Цель: поднять личный кабинет проверки моделей в локальной сети.
|
|
|
|
LAN URL:
|
|
|
|
```text
|
|
http://192.168.220.91:8765/tools/model-chat/
|
|
```
|
|
|
|
## Deploy
|
|
|
|
Preflight before deployment:
|
|
|
|
```powershell
|
|
powershell -ExecutionPolicy Bypass -File scripts\preflight_model_chat_ui.ps1 `
|
|
-DockerHost ssh://docker-gpu `
|
|
-SshTarget docker-gpu `
|
|
-CheckLive
|
|
```
|
|
|
|
Fast local-only preflight:
|
|
|
|
```powershell
|
|
powershell -ExecutionPolicy Bypass -File scripts\preflight_model_chat_ui.ps1 -SkipDockerConfig
|
|
```
|
|
|
|
```powershell
|
|
powershell -ExecutionPolicy Bypass -File scripts/deploy_model_chat_ui.ps1 `
|
|
-DockerHost ssh://docker-gpu `
|
|
-SshTarget docker-gpu
|
|
```
|
|
|
|
Скрипт:
|
|
|
|
- собирает небольшой архив приложения без тяжелых `models/`;
|
|
- копирует его на Windows GPU-хост;
|
|
- распаковывает в `Z:\LLM\model-chat-app`;
|
|
- запускает compose-сервис `llm-model-chat-ui`;
|
|
- открывает Windows Firewall для TCP `8765`.
|
|
|
|
## Health
|
|
|
|
```powershell
|
|
Invoke-RestMethod http://192.168.220.91:8765/api/health
|
|
Invoke-RestMethod http://192.168.220.91:8765/api/catalog
|
|
Invoke-RestMethod http://192.168.220.91:8765/api/model-services
|
|
python scripts\plan_model_services.py
|
|
python scripts\generate_model_chat_status.py
|
|
```
|
|
|
|
The status generator writes a compact Markdown snapshot to:
|
|
|
|
```text
|
|
reports/model-chat/status.md
|
|
```
|
|
|
|
`/api/health` includes:
|
|
|
|
- endpoint status for vLLM, llama.cpp, translation, audio, and video service ports;
|
|
- selected route for each plugin;
|
|
- GPU profile readiness with missing services to start and conflicting services to stop;
|
|
- latest Model Chat UI preflight status from `reports/model-chat/preflight.json`;
|
|
- local storage status for every registered model;
|
|
- service plan with compose/deploy hints.
|
|
|
|
Service states:
|
|
|
|
- `online`: endpoint is running and reports the expected served model name;
|
|
- `ready_to_start`: model files are present, but the service is not online;
|
|
- `blocked`: local model files are missing or incomplete.
|
|
|
|
## Service Control
|
|
|
|
The UI has a `service` panel for the selected model. It calls `POST /api/service-control`:
|
|
|
|
```json
|
|
{
|
|
"model_id": "whisper-large-v3-turbo",
|
|
"action": "status"
|
|
}
|
|
```
|
|
|
|
Allowed actions are `start`, `stop`, `restart`, and `status`.
|
|
|
|
If the UI container has no Docker CLI or SSH key, use the operator script from the project folder:
|
|
|
|
```powershell
|
|
powershell -ExecutionPolicy Bypass -File scripts\manage_model_service.ps1 -Service audio-api -Action status
|
|
powershell -ExecutionPolicy Bypass -File scripts\manage_model_service.ps1 -Service audio-api -Action stop
|
|
powershell -ExecutionPolicy Bypass -File scripts\manage_model_service.ps1 -Service translation-api -Action start
|
|
```
|
|
|
|
Use `stop` on heavy services before starting another large model when VRAM is low.
|
|
|
|
## GPU Profiles
|
|
|
|
Use GPU profiles instead of manual container juggling when switching between heavy models:
|
|
|
|
```powershell
|
|
powershell -ExecutionPolicy Bypass -File scripts\switch_gpu_profile.ps1 -Profile default
|
|
powershell -ExecutionPolicy Bypass -File scripts\switch_gpu_profile.ps1 -Profile text
|
|
powershell -ExecutionPolicy Bypass -File scripts\switch_gpu_profile.ps1 -Profile audio
|
|
powershell -ExecutionPolicy Bypass -File scripts\switch_gpu_profile.ps1 -Profile video
|
|
powershell -ExecutionPolicy Bypass -File scripts\switch_gpu_profile.ps1 -Profile image
|
|
powershell -ExecutionPolicy Bypass -File scripts\switch_gpu_profile.ps1 -Profile gguf-1c
|
|
```
|
|
|
|
Profile definitions live in:
|
|
|
|
```text
|
|
config/gpu_profiles.json
|
|
```
|
|
|
|
The UI catalog, `/api/health` profile readiness, and `scripts\switch_gpu_profile.ps1` read the same file.
|
|
Validate it before deploy:
|
|
|
|
```powershell
|
|
python scripts\validate_gpu_profiles.py --print
|
|
```
|
|
|
|
Validate the UI deployment archive without uploading it:
|
|
|
|
```powershell
|
|
powershell -ExecutionPolicy Bypass -File scripts\deploy_model_chat_ui.ps1 -ArchiveOnly
|
|
```
|
|
|
|
Dry-run the plan without touching containers:
|
|
|
|
```powershell
|
|
powershell -ExecutionPolicy Bypass -File scripts\switch_gpu_profile.ps1 -Profile video -PlanOnly
|
|
```
|
|
|
|
Profiles:
|
|
|
|
- `default` / `text`: vLLM text, translation, audio, and UI; video, image, and llama.cpp stopped.
|
|
- `audio`: audio, translation, vLLM text, and UI; video, image, and llama.cpp stopped.
|
|
- `video`: translation, video, and UI; vLLM text, audio, image, and llama.cpp stopped to free VRAM.
|
|
- `image`: translation, image, and UI; vLLM text, audio, video, and llama.cpp stopped to free VRAM.
|
|
- `gguf-1c`: llama.cpp, translation, and UI; vLLM text, audio, video, and image stopped.
|
|
|
|
The Model Chat UI also shows the recommended profile command for the selected plugin.
|
|
The profile hint shows `status: ready` when the current containers already match the profile.
|
|
Otherwise it lists services that should be started and services that should be stopped.
|
|
For `video` and `image`, the service starts quickly but the first request can still spend several minutes loading
|
|
the model into GPU memory.
|
|
|
|
## Runtime Profiles
|
|
|
|
Runtime profiles choose the execution host and endpoint used by the chat UI:
|
|
|
|
```text
|
|
config/runtime_profiles.json
|
|
```
|
|
|
|
Profiles:
|
|
|
|
- `gpu-fast`: default interactive profile on `docker-gpu.cin.su`; uses RTX 4090 endpoints.
|
|
- `cpu-test`: benchmark/fallback profile on `docker-test.cin.su`; currently maps Qwen3-Coder Q6 to `http://docker-test.cin.su:18086` with served model `qwen3-coder-1c-q6-cpu`.
|
|
- `background`: batch profile for downloads, RAG indexing and conversions; not intended for direct chat.
|
|
|
|
The UI applies `model_overrides` from the selected runtime profile. This allows the same registry model
|
|
to use a different endpoint or served model name on another host.
|
|
|
|
The service panel has `Benchmark GPU / CPU`. It calls `POST /api/benchmark/runtime`, runs
|
|
`scripts/benchmark_runtime_profiles.py`, and stores the raw report in:
|
|
|
|
```text
|
|
/reports/benchmarks/runtime-profiles-<model-id>-<timestamp>.json
|
|
```
|
|
|
|
The UI loads recent benchmark history for the currently selected model/plugin through
|
|
`GET /api/benchmark/history?model_id=<model-id>&plugin=<plugin-id>`.
|
|
When both `gpu-fast` and `cpu-test` succeed, the report includes `speedup.gpu_vs_cpu_ratio`.
|
|
Use larger generation limits, for example 192-384 tokens, for representative GPU/CPU ratios;
|
|
very short runs include more startup and request overhead.
|
|
The UI has a separate `Benchmark tokens` field, default `384`, so chat generation limits do not
|
|
accidentally make CPU comparison runs too long.
|
|
Before starting the long benchmark request, the UI performs a quick preflight for both `gpu-fast`
|
|
and `cpu-test`; if either served model is missing, the benchmark is not started.
|
|
|
|
```text
|
|
POST /api/benchmark/preflight
|
|
```
|
|
|
|
CLI check:
|
|
|
|
```powershell
|
|
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\check_runtime_preflight.ps1
|
|
```
|
|
|
|
Each history row links to the raw JSON report through:
|
|
|
|
```text
|
|
/api/benchmark/report?name=<runtime-profiles-report.json>
|
|
```
|
|
|
|
It also links to a generated Markdown summary:
|
|
|
|
```text
|
|
/api/benchmark/report.md?name=<runtime-profiles-report.json>
|
|
```
|
|
|
|
For `cpu-test`, start the heavyweight llama.cpp CPU server before sending chat requests:
|
|
|
|
```powershell
|
|
powershell -ExecutionPolicy Bypass -File scripts\manage_cpu_q6_service.ps1 -Action start
|
|
powershell -ExecutionPolicy Bypass -File scripts\manage_cpu_q6_service.ps1 -Action status
|
|
powershell -ExecutionPolicy Bypass -File scripts\manage_cpu_q6_service.ps1 -Action logs
|
|
```
|
|
|
|
For `gpu-fast` Qwen3-Coder Q6 checks on port `8081`, use the GPU launcher. Stop image/video/vLLM
|
|
first if VRAM is tight:
|
|
|
|
```powershell
|
|
powershell -ExecutionPolicy Bypass -File scripts\switch_gpu_profile.ps1 -Profile gguf-1c
|
|
powershell -ExecutionPolicy Bypass -File scripts\manage_gpu_q6_service.ps1 -Action start
|
|
powershell -ExecutionPolicy Bypass -File scripts\manage_gpu_q6_service.ps1 -Action status
|
|
powershell -ExecutionPolicy Bypass -File scripts\manage_gpu_q6_service.ps1 -Action logs
|
|
```
|
|
|
|
After you convert a trained PEFT LoRA adapter to GGUF for `llama.cpp`, you can start the same
|
|
service with the adapter applied:
|
|
|
|
```powershell
|
|
powershell -ExecutionPolicy Bypass -File scripts\convert_1c_lora_to_gguf_gpu.ps1
|
|
|
|
powershell -ExecutionPolicy Bypass -File scripts\manage_gpu_q6_service.ps1 `
|
|
-Action start `
|
|
-LoraPath /models/adapters/1c/qwen3-coder-30b-a3b-1c-lora-v1.gguf
|
|
```
|
|
|
|
Stop it after benchmarks on the shared host if it is no longer needed:
|
|
|
|
```powershell
|
|
powershell -ExecutionPolicy Bypass -File scripts\manage_cpu_q6_service.ps1 -Action stop
|
|
powershell -ExecutionPolicy Bypass -File scripts\manage_gpu_q6_service.ps1 -Action stop
|
|
```
|
|
|
|
## Containers
|
|
|
|
```powershell
|
|
docker -H ssh://docker-gpu ps
|
|
docker -H ssh://docker-gpu logs --tail 100 llm-model-chat-ui
|
|
docker -H ssh://docker-gpu logs --tail 100 llm-vllm-text
|
|
docker -H ssh://docker-gpu logs --tail 100 llm-transformers-translation
|
|
```
|
|
|
|
## Transformers Plugin Services
|
|
|
|
Translation, audio, video, and image services use the small OpenAI-like server in
|
|
`scripts/transformers_plugin_server.py`.
|
|
|
|
```powershell
|
|
powershell -ExecutionPolicy Bypass -File scripts\deploy_transformers_service.ps1 -Plugin translation
|
|
powershell -ExecutionPolicy Bypass -File scripts\deploy_transformers_service.ps1 -Plugin audio
|
|
powershell -ExecutionPolicy Bypass -File scripts\deploy_transformers_service.ps1 -Plugin video
|
|
powershell -ExecutionPolicy Bypass -File scripts\deploy_transformers_service.ps1 -Plugin image
|
|
```
|
|
|
|
By default the deploy script also refreshes `Z:\LLM\model-chat-app` before restarting the selected
|
|
service, so changes in `scripts/transformers_plugin_server.py` are applied to the mounted `/app`
|
|
directory. Use `-NoSyncApp` only for a fast container restart when the app files are already current.
|
|
|
|
Ports:
|
|
|
|
- translation: `8010`
|
|
- audio: `8020`
|
|
- video: `8030`
|
|
- image: `8040`
|
|
|
|
The translation service exposes `/health`, `/v1/models`, and `/v1/chat/completions`.
|
|
The audio service exposes `/health`, `/v1/models`, and `/v1/audio/transcriptions`; in the UI,
|
|
select plugin `Звук`, choose an audio file, then press `Отправить`.
|
|
The video service exposes `/health`, `/v1/models`, and `/v1/vision/analyze`; in the UI,
|
|
select plugin `Видео`, choose an image, enter the question in the prompt box, then press `Отправить`.
|
|
The image service exposes `/health`, `/v1/models`, `/v1/images/generations`, `/v1/images/edits`,
|
|
and async job endpoints under `/v1/images/jobs`. In the UI, select plugin `Фото`, choose
|
|
an image model, choose `generate` or `edit / inpaint`, enter the prompt, then press `Отправить`.
|
|
By default `LOAD_ON_START=1` and `BACKGROUND_LOAD_ON_START=1` for image generation, so the service
|
|
opens HTTP quickly and loads the SDXL base model in the background. The first request can show
|
|
`loading_model` for several minutes. Switching between SDXL base and inpaint may unload the other
|
|
pipeline to keep VRAM available.
|
|
Stop the large text vLLM container before loading another large model if VRAM is tight.
|
|
|
|
The UI can select `Qwen Image Edit`, but the image endpoint must actually serve
|
|
`qwen-image-edit`; otherwise `/api/image/submit` rejects the job instead of silently using SDXL.
|
|
To test Qwen Image Edit as a single-heavy-model experiment, restart the image service with the Qwen
|
|
env file:
|
|
|
|
```powershell
|
|
docker -H ssh://docker-gpu compose `
|
|
--env-file core/deploy/docker-gpu/transformers/image.qwen-edit.env.example `
|
|
-f core/deploy/docker-gpu/transformers/image.compose.yaml up -d
|
|
```
|
|
|
|
Restore the verified SDXL service with:
|
|
|
|
```powershell
|
|
powershell -ExecutionPolicy Bypass -File scripts\deploy_transformers_service.ps1 -Plugin image
|
|
```
|
|
|
|
Verified Qwen Image Edit notes from `2026-06-20`:
|
|
|
|
- model files are present under `Z:\LLM\models\image\qwen-image-edit`;
|
|
- `diffusers` detects `QwenImageEditPipeline` and the service can expose `/v1/models` as
|
|
`qwen-image-edit`;
|
|
- pipeline load completed in about `31 s` with CPU offload on RTX 4090;
|
|
- a 512x512 edit job with `1` inference step did not finish within `1800 s`, so Qwen Image Edit is
|
|
not practical on the current 24 GB GPU profile without a quantized/optimized runtime or a larger
|
|
GPU;
|
|
- keep SDXL as the default verified image service for now.
|
|
|
|
Smoke test image jobs through the Model Chat proxy:
|
|
|
|
```powershell
|
|
python scripts\smoke_image_jobs.py --operation generate --steps 4
|
|
python scripts\smoke_image_jobs.py --operation edit --steps 4
|
|
python scripts\smoke_image_jobs.py --operation edit --steps 40 --cancel --cancel-after 1
|
|
python scripts\smoke_image_jobs.py --model-id qwen-image-edit --model-mode qwen-image-edit --model qwen-image-edit --operation edit --steps 4
|
|
```
|
|
|
|
Audio and video smoke checks generate small synthetic inputs locally and send them through the
|
|
Model Chat UI proxy:
|
|
|
|
```powershell
|
|
powershell -ExecutionPolicy Bypass -File scripts\switch_gpu_profile.ps1 -Profile audio
|
|
python scripts\smoke_audio_transcription.py
|
|
|
|
powershell -ExecutionPolicy Bypass -File scripts\switch_gpu_profile.ps1 -Profile video
|
|
python scripts\smoke_video_analysis.py
|
|
```
|
|
|
|
When the matching service is intentionally stopped, use `--allow-unavailable` for a code-path check
|
|
that does not fail the local validation run.
|
|
|
|
Verified on `2026-06-20` with RTX 4090:
|
|
|
|
- generate, SDXL base, 512x512, 4 steps: completed in `2194 ms` after warmup;
|
|
- edit / inpaint, SDXL inpaint, 512x512, 4 steps: completed in `180649 ms` including first inpaint model load;
|
|
- generate, SDXL base, 512x512, 1 step: completed in `86866 ms` after image service restart and warmup;
|
|
- generated artifacts are served through `/generated-images/<date>/<file>.png`.
|
|
|
|
Direct artifact check uses `GET`; this minimal HTTP server does not implement `HEAD` for generated files.
|
|
|
|
## Model Downloads On GPU Host
|
|
|
|
Use this when model files should be written directly to `Z:\LLM\models` on the GPU host:
|
|
|
|
For large Hugging Face shard files, prefer the explicit range downloader. It writes final files directly
|
|
to `/models/...`, resumes by local file size, and avoids stale `.cache/huggingface/download/*.incomplete`
|
|
files left by interrupted Xet/snapshot downloads.
|
|
|
|
```powershell
|
|
powershell -ExecutionPolicy Bypass -File scripts\download_hf_range_gpu.ps1 `
|
|
-CardId qwen2_5-vl-7b-instruct `
|
|
-LocalDir /models/video/qwen2.5-vl-7b-instruct `
|
|
-AllowFile "model-00001-of-00005.safetensors,model-00002-of-00005.safetensors,model-00003-of-00005.safetensors,model-00004-of-00005.safetensors,model-00005-of-00005.safetensors" `
|
|
-Detached `
|
|
-ContainerName llm-hf-range-qwen-vl `
|
|
-ChunkSize 256mb
|
|
```
|
|
|
|
The registry/snapshot downloader is still useful for dry-run planning and small metadata files:
|
|
|
|
```powershell
|
|
powershell -ExecutionPolicy Bypass -File scripts\download_missing_hf_models_gpu.ps1 -DryRun
|
|
|
|
docker -H ssh://docker-gpu rm -f llm-model-download
|
|
powershell -ExecutionPolicy Bypass -File scripts\download_missing_hf_models_gpu.ps1 `
|
|
-Detached `
|
|
-ContainerName llm-model-download
|
|
```
|
|
|
|
Monitor:
|
|
|
|
```powershell
|
|
docker -H ssh://docker-gpu ps
|
|
docker -H ssh://docker-gpu logs --tail 100 llm-model-download
|
|
Invoke-RestMethod http://192.168.220.91:8765/api/model-services
|
|
```
|
|
|
|
## Notes
|
|
|
|
- UI is served by `scripts/model_chat_server.py`.
|
|
- Inference endpoint defaults to `http://docker-gpu.cin.su:8000`.
|
|
- Use `vllm/vllm-openai:v0.10.2` for the UI image because it already contains Python and is verified on this host.
|
|
- `vllm/vllm-openai:latest` passes CUDA after driver `595.97`, but vLLM `0.23.0` currently fails on Docker Desktop/WSL with `UVA is not available`.
|