Initial SQL-only 1C adapter baseline

2026-07-22 03:03:47 +03:00
commit e2503b77e7
545 changed files with 184711 additions and 0 deletions
@@ -0,0 +1,388 @@
+# Model Chat UI Service
+
+Цель: поднять личный кабинет проверки моделей в локальной сети.
+
+LAN URL:
+
+```text
+http://192.168.220.91:8765/tools/model-chat/
+```
+
+## Deploy
+
+Preflight before deployment:
+
+```powershell
+powershell -ExecutionPolicy Bypass -File scripts\preflight_model_chat_ui.ps1 `
+  -DockerHost ssh://docker-gpu `
+  -SshTarget docker-gpu `
+  -CheckLive
+```
+
+Fast local-only preflight:
+
+```powershell
+powershell -ExecutionPolicy Bypass -File scripts\preflight_model_chat_ui.ps1 -SkipDockerConfig
+```
+
+```powershell
+powershell -ExecutionPolicy Bypass -File scripts/deploy_model_chat_ui.ps1 `
+  -DockerHost ssh://docker-gpu `
+  -SshTarget docker-gpu
+```
+
+Скрипт:
+
+- собирает небольшой архив приложения без тяжелых `models/`;
+- копирует его на Windows GPU-хост;
+- распаковывает в `Z:\LLM\model-chat-app`;
+- запускает compose-сервис `llm-model-chat-ui`;
+- открывает Windows Firewall для TCP `8765`.
+
+## Health
+
+```powershell
+Invoke-RestMethod http://192.168.220.91:8765/api/health
+Invoke-RestMethod http://192.168.220.91:8765/api/catalog
+Invoke-RestMethod http://192.168.220.91:8765/api/model-services
+python scripts\plan_model_services.py
+python scripts\generate_model_chat_status.py
+```
+
+The status generator writes a compact Markdown snapshot to:
+
+```text
+reports/model-chat/status.md
+```
+
+`/api/health` includes:
+
+- endpoint status for vLLM, llama.cpp, translation, audio, and video service ports;
+- selected route for each plugin;
+- GPU profile readiness with missing services to start and conflicting services to stop;
+- latest Model Chat UI preflight status from `reports/model-chat/preflight.json`;
+- local storage status for every registered model;
+- service plan with compose/deploy hints.
+
+Service states:
+
+- `online`: endpoint is running and reports the expected served model name;
+- `ready_to_start`: model files are present, but the service is not online;
+- `blocked`: local model files are missing or incomplete.
+
+## Service Control
+
+The UI has a `service` panel for the selected model. It calls `POST /api/service-control`:
+
+```json
+{
+  "model_id": "whisper-large-v3-turbo",
+  "action": "status"
+}
+```
+
+Allowed actions are `start`, `stop`, `restart`, and `status`.
+
+If the UI container has no Docker CLI or SSH key, use the operator script from the project folder:
+
+```powershell
+powershell -ExecutionPolicy Bypass -File scripts\manage_model_service.ps1 -Service audio-api -Action status
+powershell -ExecutionPolicy Bypass -File scripts\manage_model_service.ps1 -Service audio-api -Action stop
+powershell -ExecutionPolicy Bypass -File scripts\manage_model_service.ps1 -Service translation-api -Action start
+```
+
+Use `stop` on heavy services before starting another large model when VRAM is low.
+
+## GPU Profiles
+
+Use GPU profiles instead of manual container juggling when switching between heavy models:
+
+```powershell
+powershell -ExecutionPolicy Bypass -File scripts\switch_gpu_profile.ps1 -Profile default
+powershell -ExecutionPolicy Bypass -File scripts\switch_gpu_profile.ps1 -Profile text
+powershell -ExecutionPolicy Bypass -File scripts\switch_gpu_profile.ps1 -Profile audio
+powershell -ExecutionPolicy Bypass -File scripts\switch_gpu_profile.ps1 -Profile video
+powershell -ExecutionPolicy Bypass -File scripts\switch_gpu_profile.ps1 -Profile image
+powershell -ExecutionPolicy Bypass -File scripts\switch_gpu_profile.ps1 -Profile gguf-1c
+```
+
+Profile definitions live in:
+
+```text
+config/gpu_profiles.json
+```
+
+The UI catalog, `/api/health` profile readiness, and `scripts\switch_gpu_profile.ps1` read the same file.
+Validate it before deploy:
+
+```powershell
+python scripts\validate_gpu_profiles.py --print
+```
+
+Validate the UI deployment archive without uploading it:
+
+```powershell
+powershell -ExecutionPolicy Bypass -File scripts\deploy_model_chat_ui.ps1 -ArchiveOnly
+```
+
+Dry-run the plan without touching containers:
+
+```powershell
+powershell -ExecutionPolicy Bypass -File scripts\switch_gpu_profile.ps1 -Profile video -PlanOnly
+```
+
+Profiles:
+
+- `default` / `text`: vLLM text, translation, audio, and UI; video, image, and llama.cpp stopped.
+- `audio`: audio, translation, vLLM text, and UI; video, image, and llama.cpp stopped.
+- `video`: translation, video, and UI; vLLM text, audio, image, and llama.cpp stopped to free VRAM.
+- `image`: translation, image, and UI; vLLM text, audio, video, and llama.cpp stopped to free VRAM.
+- `gguf-1c`: llama.cpp, translation, and UI; vLLM text, audio, video, and image stopped.
+
+The Model Chat UI also shows the recommended profile command for the selected plugin.
+The profile hint shows `status: ready` when the current containers already match the profile.
+Otherwise it lists services that should be started and services that should be stopped.
+For `video` and `image`, the service starts quickly but the first request can still spend several minutes loading
+the model into GPU memory.
+
+## Runtime Profiles
+
+Runtime profiles choose the execution host and endpoint used by the chat UI:
+
+```text
+config/runtime_profiles.json
+```
+
+Profiles:
+
+- `gpu-fast`: default interactive profile on `docker-gpu.cin.su`; uses RTX 4090 endpoints.
+- `cpu-test`: benchmark/fallback profile on `docker-test.cin.su`; currently maps Qwen3-Coder Q6 to `http://docker-test.cin.su:18086` with served model `qwen3-coder-1c-q6-cpu`.
+- `background`: batch profile for downloads, RAG indexing and conversions; not intended for direct chat.
+
+The UI applies `model_overrides` from the selected runtime profile. This allows the same registry model
+to use a different endpoint or served model name on another host.
+
+The service panel has `Benchmark GPU / CPU`. It calls `POST /api/benchmark/runtime`, runs
+`scripts/benchmark_runtime_profiles.py`, and stores the raw report in:
+
+```text
+/reports/benchmarks/runtime-profiles-<model-id>-<timestamp>.json
+```
+
+The UI loads recent benchmark history for the currently selected model/plugin through
+`GET /api/benchmark/history?model_id=<model-id>&plugin=<plugin-id>`.
+When both `gpu-fast` and `cpu-test` succeed, the report includes `speedup.gpu_vs_cpu_ratio`.
+Use larger generation limits, for example 192-384 tokens, for representative GPU/CPU ratios;
+very short runs include more startup and request overhead.
+The UI has a separate `Benchmark tokens` field, default `384`, so chat generation limits do not
+accidentally make CPU comparison runs too long.
+Before starting the long benchmark request, the UI performs a quick preflight for both `gpu-fast`
+and `cpu-test`; if either served model is missing, the benchmark is not started.
+
+```text
+POST /api/benchmark/preflight
+```
+
+CLI check:
+
+```powershell
+powershell -NoProfile -ExecutionPolicy Bypass -File scripts\check_runtime_preflight.ps1
+```
+
+Each history row links to the raw JSON report through:
+
+```text
+/api/benchmark/report?name=<runtime-profiles-report.json>
+```
+
+It also links to a generated Markdown summary:
+
+```text
+/api/benchmark/report.md?name=<runtime-profiles-report.json>
+```
+
+For `cpu-test`, start the heavyweight llama.cpp CPU server before sending chat requests:
+
+```powershell
+powershell -ExecutionPolicy Bypass -File scripts\manage_cpu_q6_service.ps1 -Action start
+powershell -ExecutionPolicy Bypass -File scripts\manage_cpu_q6_service.ps1 -Action status
+powershell -ExecutionPolicy Bypass -File scripts\manage_cpu_q6_service.ps1 -Action logs
+```
+
+For `gpu-fast` Qwen3-Coder Q6 checks on port `8081`, use the GPU launcher. Stop image/video/vLLM
+first if VRAM is tight:
+
+```powershell
+powershell -ExecutionPolicy Bypass -File scripts\switch_gpu_profile.ps1 -Profile gguf-1c
+powershell -ExecutionPolicy Bypass -File scripts\manage_gpu_q6_service.ps1 -Action start
+powershell -ExecutionPolicy Bypass -File scripts\manage_gpu_q6_service.ps1 -Action status
+powershell -ExecutionPolicy Bypass -File scripts\manage_gpu_q6_service.ps1 -Action logs
+```
+
+After you convert a trained PEFT LoRA adapter to GGUF for `llama.cpp`, you can start the same
+service with the adapter applied:
+
+```powershell
+powershell -ExecutionPolicy Bypass -File scripts\convert_1c_lora_to_gguf_gpu.ps1
+
+powershell -ExecutionPolicy Bypass -File scripts\manage_gpu_q6_service.ps1 `
+  -Action start `
+  -LoraPath /models/adapters/1c/qwen3-coder-30b-a3b-1c-lora-v1.gguf
+```
+
+Stop it after benchmarks on the shared host if it is no longer needed:
+
+```powershell
+powershell -ExecutionPolicy Bypass -File scripts\manage_cpu_q6_service.ps1 -Action stop
+powershell -ExecutionPolicy Bypass -File scripts\manage_gpu_q6_service.ps1 -Action stop
+```
+
+## Containers
+
+```powershell
+docker -H ssh://docker-gpu ps
+docker -H ssh://docker-gpu logs --tail 100 llm-model-chat-ui
+docker -H ssh://docker-gpu logs --tail 100 llm-vllm-text
+docker -H ssh://docker-gpu logs --tail 100 llm-transformers-translation
+```
+
+## Transformers Plugin Services
+
+Translation, audio, video, and image services use the small OpenAI-like server in
+`scripts/transformers_plugin_server.py`.
+
+```powershell
+powershell -ExecutionPolicy Bypass -File scripts\deploy_transformers_service.ps1 -Plugin translation
+powershell -ExecutionPolicy Bypass -File scripts\deploy_transformers_service.ps1 -Plugin audio
+powershell -ExecutionPolicy Bypass -File scripts\deploy_transformers_service.ps1 -Plugin video
+powershell -ExecutionPolicy Bypass -File scripts\deploy_transformers_service.ps1 -Plugin image
+```
+
+By default the deploy script also refreshes `Z:\LLM\model-chat-app` before restarting the selected
+service, so changes in `scripts/transformers_plugin_server.py` are applied to the mounted `/app`
+directory. Use `-NoSyncApp` only for a fast container restart when the app files are already current.
+
+Ports:
+
+- translation: `8010`
+- audio: `8020`
+- video: `8030`
+- image: `8040`
+
+The translation service exposes `/health`, `/v1/models`, and `/v1/chat/completions`.
+The audio service exposes `/health`, `/v1/models`, and `/v1/audio/transcriptions`; in the UI,
+select plugin `Звук`, choose an audio file, then press `Отправить`.
+The video service exposes `/health`, `/v1/models`, and `/v1/vision/analyze`; in the UI,
+select plugin `Видео`, choose an image, enter the question in the prompt box, then press `Отправить`.
+The image service exposes `/health`, `/v1/models`, `/v1/images/generations`, `/v1/images/edits`,
+and async job endpoints under `/v1/images/jobs`. In the UI, select plugin `Фото`, choose
+an image model, choose `generate` or `edit / inpaint`, enter the prompt, then press `Отправить`.
+By default `LOAD_ON_START=1` and `BACKGROUND_LOAD_ON_START=1` for image generation, so the service
+opens HTTP quickly and loads the SDXL base model in the background. The first request can show
+`loading_model` for several minutes. Switching between SDXL base and inpaint may unload the other
+pipeline to keep VRAM available.
+Stop the large text vLLM container before loading another large model if VRAM is tight.
+
+The UI can select `Qwen Image Edit`, but the image endpoint must actually serve
+`qwen-image-edit`; otherwise `/api/image/submit` rejects the job instead of silently using SDXL.
+To test Qwen Image Edit as a single-heavy-model experiment, restart the image service with the Qwen
+env file:
+
+```powershell
+docker -H ssh://docker-gpu compose `
+  --env-file core/deploy/docker-gpu/transformers/image.qwen-edit.env.example `
+  -f core/deploy/docker-gpu/transformers/image.compose.yaml up -d
+```
+
+Restore the verified SDXL service with:
+
+```powershell
+powershell -ExecutionPolicy Bypass -File scripts\deploy_transformers_service.ps1 -Plugin image
+```
+
+Verified Qwen Image Edit notes from `2026-06-20`:
+
+- model files are present under `Z:\LLM\models\image\qwen-image-edit`;
+- `diffusers` detects `QwenImageEditPipeline` and the service can expose `/v1/models` as
+  `qwen-image-edit`;
+- pipeline load completed in about `31 s` with CPU offload on RTX 4090;
+- a 512x512 edit job with `1` inference step did not finish within `1800 s`, so Qwen Image Edit is
+  not practical on the current 24 GB GPU profile without a quantized/optimized runtime or a larger
+  GPU;
+- keep SDXL as the default verified image service for now.
+
+Smoke test image jobs through the Model Chat proxy:
+
+```powershell
+python scripts\smoke_image_jobs.py --operation generate --steps 4
+python scripts\smoke_image_jobs.py --operation edit --steps 4
+python scripts\smoke_image_jobs.py --operation edit --steps 40 --cancel --cancel-after 1
+python scripts\smoke_image_jobs.py --model-id qwen-image-edit --model-mode qwen-image-edit --model qwen-image-edit --operation edit --steps 4
+```
+
+Audio and video smoke checks generate small synthetic inputs locally and send them through the
+Model Chat UI proxy:
+
+```powershell
+powershell -ExecutionPolicy Bypass -File scripts\switch_gpu_profile.ps1 -Profile audio
+python scripts\smoke_audio_transcription.py
+
+powershell -ExecutionPolicy Bypass -File scripts\switch_gpu_profile.ps1 -Profile video
+python scripts\smoke_video_analysis.py
+```
+
+When the matching service is intentionally stopped, use `--allow-unavailable` for a code-path check
+that does not fail the local validation run.
+
+Verified on `2026-06-20` with RTX 4090:
+
+- generate, SDXL base, 512x512, 4 steps: completed in `2194 ms` after warmup;
+- edit / inpaint, SDXL inpaint, 512x512, 4 steps: completed in `180649 ms` including first inpaint model load;
+- generate, SDXL base, 512x512, 1 step: completed in `86866 ms` after image service restart and warmup;
+- generated artifacts are served through `/generated-images/<date>/<file>.png`.
+
+Direct artifact check uses `GET`; this minimal HTTP server does not implement `HEAD` for generated files.
+
+## Model Downloads On GPU Host
+
+Use this when model files should be written directly to `Z:\LLM\models` on the GPU host:
+
+For large Hugging Face shard files, prefer the explicit range downloader. It writes final files directly
+to `/models/...`, resumes by local file size, and avoids stale `.cache/huggingface/download/*.incomplete`
+files left by interrupted Xet/snapshot downloads.
+
+```powershell
+powershell -ExecutionPolicy Bypass -File scripts\download_hf_range_gpu.ps1 `
+  -CardId qwen2_5-vl-7b-instruct `
+  -LocalDir /models/video/qwen2.5-vl-7b-instruct `
+  -AllowFile "model-00001-of-00005.safetensors,model-00002-of-00005.safetensors,model-00003-of-00005.safetensors,model-00004-of-00005.safetensors,model-00005-of-00005.safetensors" `
+  -Detached `
+  -ContainerName llm-hf-range-qwen-vl `
+  -ChunkSize 256mb
+```
+
+The registry/snapshot downloader is still useful for dry-run planning and small metadata files:
+
+```powershell
+powershell -ExecutionPolicy Bypass -File scripts\download_missing_hf_models_gpu.ps1 -DryRun
+
+docker -H ssh://docker-gpu rm -f llm-model-download
+powershell -ExecutionPolicy Bypass -File scripts\download_missing_hf_models_gpu.ps1 `
+  -Detached `
+  -ContainerName llm-model-download
+```
+
+Monitor:
+
+```powershell
+docker -H ssh://docker-gpu ps
+docker -H ssh://docker-gpu logs --tail 100 llm-model-download
+Invoke-RestMethod http://192.168.220.91:8765/api/model-services
+```
+
+## Notes
+
+- UI is served by `scripts/model_chat_server.py`.
+- Inference endpoint defaults to `http://docker-gpu.cin.su:8000`.
+- Use `vllm/vllm-openai:v0.10.2` for the UI image because it already contains Python and is verified on this host.
+- `vllm/vllm-openai:latest` passes CUDA after driver `595.97`, but vLLM `0.23.0` currently fails on Docker Desktop/WSL with `UVA is not available`.