Compare commits

...
Author SHA1 Message Date
m 5fb616310d Merge remote-tracking branch 'origin/main' into codex/household-time-reply-priority
CI / test (push) Successful in 1m18s
CI / test (pull_request) Successful in 1m24s
2026-08-25 19:08:15 +03:00
m 5a9cf7a22d Preserve direct time difference replies
CI / test (push) Failing after 51s
CI / test (pull_request) Failing after 48s
2026-08-25 19:06:51 +03:00
m d649f79ca4 Merge pull request 'Preserve remembered preference identity' (#366) from codex/household-checklist-live-preference-fix into main
CI / test (push) Successful in 1m58s
Reviewed-on: #366
2026-08-25 16:00:08 +00:00
m ffce659332 Merge remote-tracking branch 'origin/main' into codex/household-checklist-live-preference-fix
CI / test (push) Successful in 1m10s
CI / test (pull_request) Successful in 1m24s
2026-08-25 18:57:06 +03:00
m 24b30bf669 Preserve remembered preference identity
CI / test (push) Failing after 51s
CI / test (pull_request) Failing after 50s
2026-08-25 18:55:33 +03:00
m 1bd9137761 Merge pull request 'Add executable 50-point household checklist' (#365) from codex/household-media-queue-isolation into main
CI / test (push) Successful in 1m39s
Reviewed-on: #365
2026-08-25 15:49:39 +00:00
m f895c5a104 Merge remote-tracking branch 'origin/main' into codex/household-media-queue-isolation
CI / test (push) Successful in 1m10s
CI / test (pull_request) Successful in 1m14s
2026-08-25 18:46:10 +03:00
m d0074709aa Add executable 50-point household checklist
CI / test (push) Failing after 49s
CI / test (pull_request) Failing after 50s
2026-08-25 18:44:49 +03:00
m 982eb6f49b Merge pull request 'Do not block image generation on general cognition' (#364) from codex/household-media-queue-isolation into main
CI / test (push) Successful in 1m24s
Reviewed-on: #364
2026-08-25 15:28:35 +00:00
m b1ee085de4 Merge remote-tracking branch 'origin/main' into codex/household-media-queue-isolation
CI / test (push) Successful in 1m12s
CI / test (pull_request) Successful in 1m15s
2026-08-25 18:25:37 +03:00
m 2c66a44910 Do not block media routes on general cognition
CI / test (push) Failing after 1m6s
CI / test (pull_request) Failing after 57s
2026-08-25 18:22:38 +03:00
m b95714a71e Merge pull request 'Isolate media cancellation and improve travel answers' (#363) from codex/household-media-queue-isolation into main
CI / test (push) Successful in 1m19s
Reviewed-on: #363
2026-08-25 15:18:22 +00:00
m 6183d811f1 Merge remote-tracking branch 'origin/main' into codex/household-media-queue-isolation
CI / test (push) Successful in 1m12s
CI / test (pull_request) Successful in 1m10s
2026-08-25 18:14:52 +03:00
m 6a2ed287ba Keep flight formatting within task scope
CI / test (push) Failing after 51s
CI / test (pull_request) Failing after 49s
2026-08-25 18:12:39 +03:00
m d342728b46 Improve nearest flight research answers
CI / test (push) Failing after 51s
2026-08-25 18:12:13 +03:00
m f121445f24 Preserve media failures and enrich nearest flight searches
CI / test (push) Failing after 52s
2026-08-25 18:08:48 +03:00
m 124871da76 Cancel only the owned ComfyUI prompt
CI / test (push) Failing after 54s
2026-08-25 18:02:59 +03:00
m 2062094228 Merge pull request 'Route creative image requests to media generation' (#362) from codex/household-scenario-audit into main
CI / test (push) Successful in 1m42s
Reviewed-on: #362
2026-08-25 14:42:39 +00:00
m 515c9713e9 Merge remote-tracking branch 'origin/main' into codex/household-scenario-audit
CI / test (push) Successful in 1m53s
CI / test (pull_request) Successful in 1m32s
2026-08-25 17:36:53 +03:00
m 14c5d9e412 Route creative image requests to media generation
CI / test (push) Failing after 1m8s
CI / test (pull_request) Failing after 1m1s
2026-08-25 17:34:05 +03:00
m 8e9429e5d0 Merge pull request 'Bind ComfyUI image provider in production' (#361) from codex/household-scenario-audit into main
CI / test (push) Successful in 1m25s
Reviewed-on: #361
2026-08-25 14:24:00 +00:00
m 6cb125001f Bind ComfyUI image provider in production
CI / test (push) Successful in 1m10s
CI / test (pull_request) Successful in 1m11s
2026-08-25 17:14:44 +03:00
m 1e7cf68317 Merge pull request 'Add ComfyUI image generation adapter' (#360) from codex/household-scenario-audit into main
CI / test (push) Successful in 1m15s
Reviewed-on: #360
2026-08-25 14:08:23 +00:00
m 2f6f6fb026 Add ComfyUI image generation adapter
CI / test (push) Successful in 1m16s
CI / test (pull_request) Failing after 57s
2026-08-25 17:01:55 +03:00
m 4618a392e2 Merge pull request 'Add replaceable image and video media routes' (#359) from codex/household-scenario-audit into main
CI / test (push) Successful in 1m19s
Reviewed-on: #359
2026-08-25 13:37:28 +00:00
m cdcbd50fe7 Test degraded media capability routing
CI / test (push) Successful in 1m15s
CI / test (pull_request) Failing after 54s
2026-08-25 16:14:28 +03:00
m 8260e69dfe Add replaceable image and video media routes
CI / test (push) Successful in 1m12s
2026-08-25 16:12:33 +03:00
m cd8ba740de Merge pull request 'Add replaceable image and video capability contracts' (#358) from codex/household-scenario-audit into main
CI / test (push) Successful in 1m45s
Merge PR #358
2026-08-25 13:05:36 +00:00
m 8173b8f9d6 Define replaceable image and video capabilities
CI / test (push) Successful in 1m32s
CI / test (pull_request) Successful in 1m31s
2026-08-25 16:01:22 +03:00
m 7d5b91fbe9 Merge pull request 'Route creative requests through cognition' (#357) from codex/household-scenario-audit into main
CI / test (push) Successful in 1m27s
Merge PR #357
2026-08-25 12:47:09 +00:00
m f9e50eba70 Remove canned creative fallback content
CI / test (push) Successful in 1m27s
CI / test (pull_request) Successful in 1m27s
2026-08-25 15:43:23 +03:00
m 29e17f1ac8 Remove canned creative object replies
CI / test (push) Successful in 1m25s
CI / test (pull_request) Successful in 1m24s
2026-08-25 15:40:29 +03:00
m ae8bf68d73 Merge pull request 'Add local horse creative fallback' (#356) from codex/household-scenario-audit into main
CI / test (push) Successful in 1m32s
Merge PR #356
2026-08-25 12:33:13 +00:00
m af146c507e Add local horse drawing fallback
CI / test (push) Successful in 1m30s
CI / test (pull_request) Successful in 1m25s
2026-08-25 15:29:25 +03:00
m 7ad7a2c8ba Merge pull request 'Add household route boundary matrix' (#355) from codex/household-scenario-audit into main
CI / test (push) Successful in 1m34s
Merge PR #355
2026-08-25 12:27:16 +00:00
m 8885a66883 Add household route boundary matrix
CI / test (push) Successful in 1m8s
CI / test (pull_request) Successful in 1m18s
2026-08-25 15:24:05 +03:00
m 2b40471e08 Merge pull request 'Add local bird creative fallback' (#354) from codex/household-scenario-audit into main
CI / test (push) Successful in 1m20s
Merge PR #354
2026-08-25 12:22:05 +00:00
m 22253caf2d Add local bird drawing fallback
CI / test (push) Successful in 1m13s
CI / test (pull_request) Successful in 1m12s
2026-08-25 15:19:03 +03:00
m f8387180eb Merge pull request 'Isolate non-travel household requests' (#353) from codex/household-scenario-audit into main
CI / test (push) Successful in 1m34s
Merge PR #353
2026-08-25 12:16:45 +00:00
m 288643f0da Cover cross-context household requests
CI / test (push) Successful in 1m25s
CI / test (pull_request) Successful in 1m26s
2026-08-25 15:13:03 +03:00
m 0662dbbb01 Merge pull request 'Expand semantic decomposition checks' (#352) from codex/household-scenario-audit into main
CI / test (push) Successful in 1m32s
Merge PR #352
2026-08-25 12:08:40 +00:00
m 1a8934f88b Keep creative requests out of travel continuation
CI / test (push) Successful in 1m25s
CI / test (pull_request) Successful in 1m22s
2026-08-25 15:04:39 +03:00
m 4cac361762 Use matched alias for task target
CI / test (push) Successful in 1m28s
CI / test (pull_request) Successful in 1m24s
2026-08-25 15:01:18 +03:00
m 515cabb5cf Expand semantic memory decomposition checks 2026-08-25 15:01:17 +03:00
m 5efa6176bb Merge pull request 'Route marketplace search and bear drawing' (#351) from codex/household-scenario-audit into main
CI / test (push) Successful in 1m36s
Merge PR #351
2026-08-25 11:55:30 +00:00
m 3515a483ca Fix generic creative target extraction
CI / test (push) Successful in 1m29s
CI / test (pull_request) Successful in 1m28s
2026-08-25 14:49:01 +03:00
m ca45df3c62 Load task aliases from semantic memory
CI / test (push) Failing after 1m0s
CI / test (pull_request) Failing after 1m2s
2026-08-25 14:48:32 +03:00
m c94a96c47e Decompose household tasks into typed atoms
CI / test (push) Successful in 1m28s
CI / test (pull_request) Successful in 1m27s
2026-08-25 14:46:43 +03:00
m 745d8b0c4d Route marketplace search and bear drawing 2026-08-25 14:46:43 +03:00
m ddc4d3bc90 Merge pull request 'Reject identity boilerplate over creative fallback' (#350) from codex/household-scenario-audit into main
CI / test (push) Successful in 1m44s
Merge PR #350
2026-08-25 11:06:26 +00:00
m 906d71b35f Reject identity boilerplate over creative fallback
CI / test (push) Successful in 1m31s
CI / test (pull_request) Successful in 1m31s
2026-08-25 14:02:42 +03:00
m beddb78469 Merge pull request 'Preserve creative fallback over conversational reflex' (#349) from codex/household-scenario-audit into main
CI / test (push) Successful in 1m34s
Merge PR #349
2026-08-25 10:57:57 +00:00
m fb968502ba Preserve creative fallback over conversational reflex
CI / test (push) Successful in 1m32s
CI / test (pull_request) Successful in 1m28s
2026-08-25 13:54:14 +03:00
m 643ce34cff Merge pull request 'Keep useful creative fallback replies' (#348) from codex/household-scenario-audit into main
CI / test (push) Successful in 1m37s
Merge PR #348
2026-08-25 10:48:58 +00:00
m cc08fb3d08 Keep useful creative fallback replies
CI / test (push) Successful in 1m35s
CI / test (pull_request) Successful in 1m32s
2026-08-25 13:45:01 +03:00
m f3bab3177d Merge pull request 'Humanize web research replies' (#347) from codex/household-scenario-audit into main
CI / test (push) Successful in 1m36s
Merge PR #347
2026-08-25 10:38:47 +00:00
m e79302c915 Humanize web research replies
CI / test (push) Successful in 1m27s
CI / test (pull_request) Successful in 1m26s
2026-08-25 13:34:48 +03:00
m d2f55bf0be Merge pull request 'Preserve assistant name in cognitive context' (#346) from codex/household-scenario-audit into main
CI / test (push) Successful in 1m19s
Merge PR #346
2026-08-25 09:53:12 +00:00
m c0208de111 Preserve assistant name in cognitive context
CI / test (push) Successful in 1m10s
CI / test (pull_request) Successful in 1m13s
2026-08-25 12:50:23 +03:00
m 56cd297ca3 Merge pull request 'Handle reasoning-only vision responses' (#345) from codex/household-scenario-audit into main
CI / test (push) Successful in 1m14s
Merge PR #345
2026-08-25 09:37:23 +00:00
m 5594525893 Handle reasoning-only vision responses
CI / test (push) Successful in 1m8s
CI / test (pull_request) Failing after 52s
2026-08-25 12:36:57 +03:00
m f91613b5d2 Merge pull request 'Add local speech, vision and OCR chat skills' (#344) from codex/household-scenario-audit into main
CI / test (push) Successful in 1m15s
2026-08-25 09:29:25 +00:00
m 78e7e3fedc Add local speech and vision chat skills
CI / test (push) Successful in 1m9s
CI / test (pull_request) Successful in 1m7s
2026-08-25 12:26:41 +03:00
m df017a349a Merge pull request 'Improve degraded household replies and route tests' (#343) from codex/household-scenario-audit into main
CI / test (push) Successful in 1m13s
2026-08-25 09:11:58 +00:00
m 8124fcbcc3 Improve degraded household replies
CI / test (push) Successful in 1m7s
CI / test (pull_request) Successful in 1m4s
2026-08-25 12:08:52 +03:00
m 8658db537d Merge pull request 'Add household scenario audit plan' (#342) from codex/household-scenario-audit into main
CI / test (push) Successful in 1m34s
2026-08-25 09:04:33 +00:00
m bd382243a0 Add household scenario audit plan
CI / test (push) Successful in 1m44s
CI / test (pull_request) Successful in 1m27s
2026-08-25 12:00:25 +03:00
m d0057f1873 Merge pull request 'Use local creative fallback when cognition is unavailable' (#341) from codex/creative-chat-routing into main
CI / test (push) Successful in 1m31s
Use local creative fallback when cognition is unavailable
2026-08-25 08:49:57 +00:00
35 changed files with 2239 additions and 67 deletions
+509
View File
@@ -0,0 +1,509 @@
package multimodal
import (
"bytes"
"context"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"io"
"log"
"mime/multipart"
"net/http"
"net/textproto"
"net/url"
"strings"
"time"
)
var ErrEmptyResult = errors.New("multimodal provider returned an empty result")
type SpeechClient struct {
BaseURL string
Client *http.Client
}
type VisionClient struct {
BaseURL string
Client *http.Client
}
type ImageGenerationClient struct {
BaseURL string
Client *http.Client
}
type VideoUnderstandingClient struct {
BaseURL string
Client *http.Client
}
type GeneratedImage struct {
DataBase64 string
SourceURL string
ContentType string
}
type ImageGenerator interface {
Generate(ctx context.Context, model, prompt, size string) (GeneratedImage, error)
}
// ComfyImageGenerationClient is the replaceable transport for a ComfyUI
// image-generation skill. The workflow is deliberately kept in this adapter:
// the core only selects a capability and supplies the user's prompt.
type ComfyImageGenerationClient struct {
BaseURL string
Client *http.Client
PollInterval time.Duration
PollTimeout time.Duration
}
func (c SpeechClient) Transcribe(ctx context.Context, filename, contentType string, data []byte) (string, error) {
if len(data) == 0 {
return "", errors.New("audio payload is empty")
}
body := &bytes.Buffer{}
writer := multipart.NewWriter(body)
header := make(textproto.MIMEHeader)
header.Set("Content-Disposition", fmt.Sprintf(`form-data; name="file"; filename="%s"`, strings.ReplaceAll(filename, `"`, "")))
header.Set("Content-Type", contentType)
part, err := writer.CreatePart(header)
if err != nil {
return "", err
}
if _, err := part.Write(data); err != nil {
return "", err
}
if err := writer.WriteField("response_format", "json"); err != nil {
return "", err
}
if err := writer.Close(); err != nil {
return "", err
}
request, err := http.NewRequestWithContext(ctx, http.MethodPost, strings.TrimRight(c.BaseURL, "/")+"/inference", body)
if err != nil {
return "", err
}
request.Header.Set("Content-Type", writer.FormDataContentType())
response, err := c.client().Do(request)
if err != nil {
return "", err
}
defer response.Body.Close()
if response.StatusCode != http.StatusOK {
return "", fmt.Errorf("speech provider status %d", response.StatusCode)
}
var result struct {
Text string `json:"text"`
}
if err := json.NewDecoder(io.LimitReader(response.Body, 1<<20)).Decode(&result); err != nil {
return "", err
}
if strings.TrimSpace(result.Text) == "" {
return "", ErrEmptyResult
}
return strings.TrimSpace(result.Text), nil
}
func (c VisionClient) Describe(ctx context.Context, model, contentType string, data []byte, instruction string) (string, error) {
if len(data) == 0 {
return "", errors.New("image payload is empty")
}
payload := struct {
Model string `json:"model"`
Messages []struct {
Role string `json:"role"`
Content []struct {
Type string `json:"type"`
Text string `json:"text,omitempty"`
ImageURL *struct {
URL string `json:"url"`
} `json:"image_url,omitempty"`
} `json:"content"`
} `json:"messages"`
MaxTokens int `json:"max_tokens"`
}{}
payload.Model = model
payload.MaxTokens = 1024
payload.Messages = append(payload.Messages, struct {
Role string `json:"role"`
Content []struct {
Type string `json:"type"`
Text string `json:"text,omitempty"`
ImageURL *struct {
URL string `json:"url"`
} `json:"image_url,omitempty"`
} `json:"content"`
}{Role: "user", Content: []struct {
Type string `json:"type"`
Text string `json:"text,omitempty"`
ImageURL *struct {
URL string `json:"url"`
} `json:"image_url,omitempty"`
}{
{Type: "text", Text: instruction},
{Type: "image_url", ImageURL: &struct {
URL string `json:"url"`
}{URL: "data:" + contentType + ";base64," + encodeBase64(data)},
}}})
encoded, err := json.Marshal(payload)
if err != nil {
return "", err
}
request, err := http.NewRequestWithContext(ctx, http.MethodPost, strings.TrimRight(c.BaseURL, "/")+"/chat/completions", bytes.NewReader(encoded))
if err != nil {
return "", err
}
request.Header.Set("Content-Type", "application/json")
response, err := c.client().Do(request)
if err != nil {
return "", err
}
defer response.Body.Close()
if response.StatusCode != http.StatusOK {
return "", fmt.Errorf("vision provider status %d", response.StatusCode)
}
var result struct {
Choices []struct {
Message struct {
Content string `json:"content"`
ReasoningContent string `json:"reasoning_content"`
} `json:"message"`
} `json:"choices"`
}
if err := json.NewDecoder(io.LimitReader(response.Body, 2<<20)).Decode(&result); err != nil {
return "", err
}
if len(result.Choices) == 0 {
return "", ErrEmptyResult
}
text := strings.TrimSpace(result.Choices[0].Message.Content)
if text == "" {
text = strings.TrimSpace(result.Choices[0].Message.ReasoningContent)
}
if text == "" {
return "", ErrEmptyResult
}
return text, nil
}
// Generate uses an OpenAI-compatible image-generation skill. The provider is
// selected outside the core by the capability binding; this adapter contains
// transport only and never decides what the user meant.
func (c ImageGenerationClient) Generate(ctx context.Context, model, prompt, size string) (GeneratedImage, error) {
if strings.TrimSpace(prompt) == "" {
return GeneratedImage{}, errors.New("image prompt is empty")
}
payload := struct {
Model string `json:"model"`
Prompt string `json:"prompt"`
Size string `json:"size,omitempty"`
ResponseFormat string `json:"response_format,omitempty"`
}{Model: model, Prompt: prompt, Size: size, ResponseFormat: "b64_json"}
encoded, err := json.Marshal(payload)
if err != nil {
return GeneratedImage{}, err
}
request, err := http.NewRequestWithContext(ctx, http.MethodPost, strings.TrimRight(c.BaseURL, "/")+"/images/generations", bytes.NewReader(encoded))
if err != nil {
return GeneratedImage{}, err
}
request.Header.Set("Content-Type", "application/json")
response, err := c.client().Do(request)
if err != nil {
return GeneratedImage{}, err
}
defer response.Body.Close()
if response.StatusCode < 200 || response.StatusCode > 299 {
return GeneratedImage{}, fmt.Errorf("image provider status %d", response.StatusCode)
}
var result struct {
Data []struct {
Base64 string `json:"b64_json"`
URL string `json:"url"`
} `json:"data"`
}
if err := json.NewDecoder(io.LimitReader(response.Body, 32<<20)).Decode(&result); err != nil {
return GeneratedImage{}, err
}
if len(result.Data) == 0 || (strings.TrimSpace(result.Data[0].Base64) == "" && strings.TrimSpace(result.Data[0].URL) == "") {
return GeneratedImage{}, ErrEmptyResult
}
return GeneratedImage{DataBase64: result.Data[0].Base64, SourceURL: result.Data[0].URL, ContentType: "image/png"}, nil
}
func (c ComfyImageGenerationClient) Generate(ctx context.Context, model, prompt, size string) (GeneratedImage, error) {
if strings.TrimSpace(prompt) == "" {
return GeneratedImage{}, errors.New("image prompt is empty")
}
width, height := imageSize(size)
workflow := map[string]any{
"3": map[string]any{"inputs": map[string]any{"seed": time.Now().UnixNano(), "steps": 8, "cfg": 1.0, "sampler_name": "euler", "scheduler": "simple", "denoise": 1.0, "model": []any{"16", 0}, "positive": []any{"6", 0}, "negative": []any{"7", 0}, "latent_image": []any{"13", 0}}, "class_type": "KSampler"},
"6": map[string]any{"inputs": map[string]any{"text": prompt, "clip": []any{"18", 0}}, "class_type": "CLIPTextEncode"},
"7": map[string]any{"inputs": map[string]any{"text": "blurry, ugly, distorted", "clip": []any{"18", 0}}, "class_type": "CLIPTextEncode"},
"8": map[string]any{"inputs": map[string]any{"samples": []any{"3", 0}, "vae": []any{"17", 0}}, "class_type": "VAEDecode"},
"9": map[string]any{"inputs": map[string]any{"filename_prefix": "domovoy", "images": []any{"8", 0}}, "class_type": "SaveImage"},
"13": map[string]any{"inputs": map[string]any{"width": width, "height": height, "batch_size": 1}, "class_type": "EmptySD3LatentImage"},
"16": map[string]any{"inputs": map[string]any{"unet_name": "z_image_turbo_bf16.safetensors", "weight_dtype": "default"}, "class_type": "UNETLoader"},
"17": map[string]any{"inputs": map[string]any{"vae_name": "ae.safetensors"}, "class_type": "VAELoader"},
"18": map[string]any{"inputs": map[string]any{"clip_name": "qwen_3_4b.safetensors", "type": "lumina2", "device": "default"}, "class_type": "CLIPLoader"},
}
payload := map[string]any{"prompt": workflow, "client_id": fmt.Sprintf("domovoy-%d", time.Now().UnixNano())}
encoded, err := json.Marshal(payload)
if err != nil {
return GeneratedImage{}, err
}
request, err := http.NewRequestWithContext(ctx, http.MethodPost, strings.TrimRight(c.BaseURL, "/")+"/prompt", bytes.NewReader(encoded))
if err != nil {
return GeneratedImage{}, err
}
request.Header.Set("Content-Type", "application/json")
response, err := c.client().Do(request)
if err != nil {
return GeneratedImage{}, err
}
defer response.Body.Close()
if response.StatusCode < 200 || response.StatusCode > 299 {
return GeneratedImage{}, fmt.Errorf("comfy provider status %d", response.StatusCode)
}
var accepted struct {
PromptID string `json:"prompt_id"`
}
if err := json.NewDecoder(io.LimitReader(response.Body, 1<<20)).Decode(&accepted); err != nil {
return GeneratedImage{}, err
}
if accepted.PromptID == "" {
return GeneratedImage{}, ErrEmptyResult
}
// A caller timeout must remove only this prompt. ComfyUI's global
// /interrupt endpoint can otherwise stop unrelated users' work and leave
// queued requests in an ambiguous state.
defer func() {
if ctx.Err() != nil {
cancelCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := c.cancelPrompt(cancelCtx, accepted.PromptID); err != nil {
// The original context error is more useful to the caller; the
// provider cleanup failure remains observable in the server log.
log.Printf("comfy prompt cleanup failed for %s: %v", accepted.PromptID, err)
}
}
}()
interval := c.PollInterval
if interval <= 0 {
interval = time.Second
}
timeout := c.PollTimeout
if timeout <= 0 {
timeout = 10 * time.Minute
}
deadline := time.NewTimer(timeout)
defer deadline.Stop()
ticker := time.NewTicker(interval)
defer ticker.Stop()
for {
image, done, err := c.historyImage(ctx, accepted.PromptID)
if err != nil {
return GeneratedImage{}, err
}
if done {
return image, nil
}
select {
case <-ctx.Done():
return GeneratedImage{}, ctx.Err()
case <-deadline.C:
return GeneratedImage{}, context.DeadlineExceeded
case <-ticker.C:
}
}
}
func (c ComfyImageGenerationClient) cancelPrompt(ctx context.Context, promptID string) error {
if strings.TrimSpace(promptID) == "" {
return errors.New("comfy prompt id is empty")
}
payload, err := json.Marshal(map[string][]string{"delete": {promptID}})
if err != nil {
return err
}
request, err := http.NewRequestWithContext(ctx, http.MethodPost, strings.TrimRight(c.BaseURL, "/")+"/queue", bytes.NewReader(payload))
if err != nil {
return err
}
request.Header.Set("Content-Type", "application/json")
response, err := c.client().Do(request)
if err != nil {
return err
}
defer response.Body.Close()
if response.StatusCode < 200 || response.StatusCode > 299 {
return fmt.Errorf("comfy prompt cleanup status %d", response.StatusCode)
}
return nil
}
func (c ComfyImageGenerationClient) historyImage(ctx context.Context, promptID string) (GeneratedImage, bool, error) {
request, err := http.NewRequestWithContext(ctx, http.MethodGet, strings.TrimRight(c.BaseURL, "/")+"/history/"+url.PathEscape(promptID), nil)
if err != nil {
return GeneratedImage{}, false, err
}
response, err := c.client().Do(request)
if err != nil {
return GeneratedImage{}, false, err
}
defer response.Body.Close()
if response.StatusCode == http.StatusNotFound {
return GeneratedImage{}, false, nil
}
if response.StatusCode < 200 || response.StatusCode > 299 {
return GeneratedImage{}, false, fmt.Errorf("comfy history status %d", response.StatusCode)
}
var history map[string]struct {
Outputs map[string]struct {
Images []struct {
Filename string `json:"filename"`
Subfolder string `json:"subfolder"`
Type string `json:"type"`
} `json:"images"`
} `json:"outputs"`
}
if err := json.NewDecoder(io.LimitReader(response.Body, 4<<20)).Decode(&history); err != nil {
return GeneratedImage{}, false, err
}
record, ok := history[promptID]
if !ok {
return GeneratedImage{}, false, nil
}
for _, output := range record.Outputs {
for _, image := range output.Images {
if image.Filename == "" {
continue
}
query := url.Values{"filename": {image.Filename}, "subfolder": {image.Subfolder}, "type": {image.Type}}
imageRequest, err := http.NewRequestWithContext(ctx, http.MethodGet, strings.TrimRight(c.BaseURL, "/")+"/view?"+query.Encode(), nil)
if err != nil {
return GeneratedImage{}, false, err
}
imageResponse, err := c.client().Do(imageRequest)
if err != nil {
return GeneratedImage{}, false, err
}
data, readErr := io.ReadAll(io.LimitReader(imageResponse.Body, 32<<20))
imageResponse.Body.Close()
if readErr != nil {
return GeneratedImage{}, false, readErr
}
if imageResponse.StatusCode < 200 || imageResponse.StatusCode > 299 || len(data) == 0 {
return GeneratedImage{}, false, ErrEmptyResult
}
contentType := imageResponse.Header.Get("Content-Type")
if contentType == "" {
contentType = "image/png"
}
return GeneratedImage{DataBase64: encodeBase64(data), ContentType: contentType}, true, nil
}
}
return GeneratedImage{}, true, ErrEmptyResult
}
func imageSize(size string) (int, int) {
parts := strings.Split(strings.TrimSpace(size), "x")
if len(parts) == 2 {
var width, height int
if _, err := fmt.Sscanf(size, "%dx%d", &width, &height); err == nil && width >= 256 && height >= 256 && width <= 2048 && height <= 2048 {
return width, height
}
}
return 512, 512
}
// Describe accepts a video skill endpoint that performs its own bounded frame
// sampling. Frame extraction is provider-owned so the core does not acquire a
// media-processing dependency or hide a policy in orchestration code.
func (c VideoUnderstandingClient) Describe(ctx context.Context, model, filename, contentType string, data []byte, instruction string) (string, error) {
if len(data) == 0 {
return "", errors.New("video payload is empty")
}
body := &bytes.Buffer{}
writer := multipart.NewWriter(body)
header := make(textproto.MIMEHeader)
header.Set("Content-Disposition", fmt.Sprintf(`form-data; name="file"; filename="%s"`, strings.ReplaceAll(filename, `"`, "")))
header.Set("Content-Type", contentType)
part, err := writer.CreatePart(header)
if err != nil {
return "", err
}
if _, err := part.Write(data); err != nil {
return "", err
}
_ = writer.WriteField("model", model)
_ = writer.WriteField("instruction", instruction)
if err := writer.Close(); err != nil {
return "", err
}
request, err := http.NewRequestWithContext(ctx, http.MethodPost, strings.TrimRight(c.BaseURL, "/")+"/understand", body)
if err != nil {
return "", err
}
request.Header.Set("Content-Type", writer.FormDataContentType())
response, err := c.client().Do(request)
if err != nil {
return "", err
}
defer response.Body.Close()
if response.StatusCode < 200 || response.StatusCode > 299 {
return "", fmt.Errorf("video provider status %d", response.StatusCode)
}
var result struct {
Text string `json:"text"`
}
if err := json.NewDecoder(io.LimitReader(response.Body, 2<<20)).Decode(&result); err != nil {
return "", err
}
if strings.TrimSpace(result.Text) == "" {
return "", ErrEmptyResult
}
return strings.TrimSpace(result.Text), nil
}
func (c SpeechClient) client() *http.Client {
if c.Client != nil {
return c.Client
}
return http.DefaultClient
}
func (c VisionClient) client() *http.Client {
if c.Client != nil {
return c.Client
}
return http.DefaultClient
}
func (c ImageGenerationClient) client() *http.Client {
if c.Client != nil {
return c.Client
}
return http.DefaultClient
}
func (c ComfyImageGenerationClient) client() *http.Client {
if c.Client != nil {
return c.Client
}
return http.DefaultClient
}
func (c VideoUnderstandingClient) client() *http.Client {
if c.Client != nil {
return c.Client
}
return http.DefaultClient
}
func encodeBase64(data []byte) string {
return base64.StdEncoding.EncodeToString(data)
}
+209
View File
@@ -0,0 +1,209 @@
package multimodal
import (
"context"
"encoding/json"
"errors"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
)
func TestSpeechClientSendsMultipartAndReturnsTranscript(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/inference" || r.Method != http.MethodPost {
t.Fatalf("unexpected speech request: %s %s", r.Method, r.URL.Path)
}
if err := r.ParseMultipartForm(1 << 20); err != nil {
t.Fatal(err)
}
file, _, err := r.FormFile("file")
if err != nil {
t.Fatal(err)
}
defer file.Close()
data, _ := io.ReadAll(file)
if string(data) != "audio" {
t.Fatalf("audio payload = %q", data)
}
_ = json.NewEncoder(w).Encode(map[string]string{"text": "распознанная фраза"})
}))
defer server.Close()
text, err := (SpeechClient{BaseURL: server.URL, Client: server.Client()}).Transcribe(context.Background(), "voice.wav", "audio/wav", []byte("audio"))
if err != nil || text != "распознанная фраза" {
t.Fatalf("transcript=%q err=%v", text, err)
}
}
func TestVisionClientSendsImageDataAndInstruction(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
var payload struct {
Messages []struct {
Content []struct {
Type string `json:"type"`
Text string `json:"text"`
ImageURL *struct {
URL string `json:"url"`
} `json:"image_url"`
} `json:"content"`
} `json:"messages"`
}
if err := json.NewDecoder(r.Body).Decode(&payload); err != nil {
t.Fatal(err)
}
if len(payload.Messages) != 1 || len(payload.Messages[0].Content) != 2 || !strings.Contains(payload.Messages[0].Content[0].Text, "Опиши") || payload.Messages[0].Content[1].ImageURL == nil || !strings.HasPrefix(payload.Messages[0].Content[1].ImageURL.URL, "data:image/png;base64,") {
t.Fatalf("unexpected vision payload: %+v", payload)
}
_ = json.NewEncoder(w).Encode(map[string]any{"choices": []any{map[string]any{"message": map[string]string{"content": "на изображении круг"}}}})
}))
defer server.Close()
text, err := (VisionClient{BaseURL: server.URL + "/v1", Client: server.Client()}).Describe(context.Background(), "vision-model", "image/png", []byte{0, 1, 2}, "Опиши изображение")
if err != nil || text != "на изображении круг" {
t.Fatalf("description=%q err=%v", text, err)
}
}
func TestVisionClientUsesReasoningContentWhenProviderOmitsContent(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_ = json.NewEncoder(w).Encode(map[string]any{"choices": []any{map[string]any{
"message": map[string]string{"content": "", "reasoning_content": "чёрный квадрат"},
}}})
}))
defer server.Close()
text, err := (VisionClient{BaseURL: server.URL, Client: server.Client()}).Describe(context.Background(), "vision-model", "image/png", []byte("image"), "Опиши изображение")
if err != nil || text != "чёрный квадрат" {
t.Fatalf("description=%q err=%v", text, err)
}
}
func TestImageGenerationClientSendsPromptAndReturnsBase64(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/images/generations" || r.Method != http.MethodPost {
t.Fatalf("unexpected image request: %s %s", r.Method, r.URL.Path)
}
var payload struct {
Model string `json:"model"`
Prompt string `json:"prompt"`
Size string `json:"size"`
}
if err := json.NewDecoder(r.Body).Decode(&payload); err != nil {
t.Fatal(err)
}
if payload.Model != "image-model" || payload.Prompt != "green bird" || payload.Size != "1024x1024" {
t.Fatalf("unexpected image payload: %+v", payload)
}
_ = json.NewEncoder(w).Encode(map[string]any{"data": []any{map[string]string{"b64_json": "aW1hZ2U="}}})
}))
defer server.Close()
image, err := (ImageGenerationClient{BaseURL: server.URL, Client: server.Client()}).Generate(context.Background(), "image-model", "green bird", "1024x1024")
if err != nil || image.DataBase64 != "aW1hZ2U=" || image.ContentType != "image/png" {
t.Fatalf("image=%+v err=%v", image, err)
}
}
func TestVideoUnderstandingClientSendsMultipartAndReturnsDescription(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/understand" || r.Method != http.MethodPost {
t.Fatalf("unexpected video request: %s %s", r.Method, r.URL.Path)
}
if err := r.ParseMultipartForm(1 << 20); err != nil {
t.Fatal(err)
}
if r.FormValue("model") != "video-model" || r.FormValue("instruction") != "describe movement" {
t.Fatalf("unexpected video fields: model=%q instruction=%q", r.FormValue("model"), r.FormValue("instruction"))
}
file, _, err := r.FormFile("file")
if err != nil {
t.Fatal(err)
}
defer file.Close()
data, _ := io.ReadAll(file)
if string(data) != "video" {
t.Fatalf("video payload=%q", data)
}
_ = json.NewEncoder(w).Encode(map[string]string{"text": "человек идёт"})
}))
defer server.Close()
text, err := (VideoUnderstandingClient{BaseURL: server.URL, Client: server.Client()}).Describe(context.Background(), "video-model", "clip.mp4", "video/mp4", []byte("video"), "describe movement")
if err != nil || text != "человек идёт" {
t.Fatalf("description=%q err=%v", text, err)
}
}
func TestComfyImageGenerationClientRunsWorkflowAndReturnsImage(t *testing.T) {
polls := 0
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/prompt":
var payload struct {
Prompt map[string]any `json:"prompt"`
}
if err := json.NewDecoder(r.Body).Decode(&payload); err != nil {
t.Fatal(err)
}
if payload.Prompt["6"].(map[string]any)["inputs"].(map[string]any)["text"] != "red bird" {
t.Fatalf("prompt was not forwarded: %+v", payload.Prompt["6"])
}
_ = json.NewEncoder(w).Encode(map[string]string{"prompt_id": "test-prompt"})
case "/history/test-prompt":
polls++
if polls == 1 {
_, _ = w.Write([]byte(`{}`))
return
}
_ = json.NewEncoder(w).Encode(map[string]any{"test-prompt": map[string]any{"outputs": map[string]any{"9": map[string]any{"images": []any{map[string]string{"filename": "out.png", "subfolder": "", "type": "output"}}}}}})
case "/view":
w.Header().Set("Content-Type", "image/png")
_, _ = w.Write([]byte("png"))
default:
http.NotFound(w, r)
}
}))
defer server.Close()
image, err := (ComfyImageGenerationClient{BaseURL: server.URL, Client: server.Client(), PollInterval: time.Millisecond, PollTimeout: time.Second}).Generate(context.Background(), "", "red bird", "256x256")
if err != nil || image.DataBase64 != "cG5n" || image.ContentType != "image/png" {
t.Fatalf("image=%+v err=%v", image, err)
}
}
func TestComfyImageGenerationClientDeletesOnlyCancelledPrompt(t *testing.T) {
var deleted []string
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/prompt":
_ = json.NewEncoder(w).Encode(map[string]string{"prompt_id": "cancel-me"})
case "/history/cancel-me":
<-r.Context().Done()
case "/queue":
var payload struct {
Delete []string `json:"delete"`
}
if err := json.NewDecoder(r.Body).Decode(&payload); err != nil {
t.Fatal(err)
}
deleted = append(deleted, payload.Delete...)
w.WriteHeader(http.StatusNoContent)
default:
http.NotFound(w, r)
}
}))
defer server.Close()
ctx, cancel := context.WithCancel(context.Background())
done := make(chan error, 1)
go func() {
_, err := (ComfyImageGenerationClient{BaseURL: server.URL, Client: server.Client(), PollInterval: time.Millisecond, PollTimeout: time.Minute}).Generate(ctx, "", "red bird", "256x256")
done <- err
}()
time.Sleep(10 * time.Millisecond)
cancel()
if err := <-done; !errors.Is(err, context.Canceled) {
t.Fatalf("err=%v", err)
}
if len(deleted) != 1 || deleted[0] != "cancel-me" {
t.Fatalf("deleted=%v", deleted)
}
}
+21
View File
@@ -7,6 +7,7 @@ import (
"errors" "errors"
"fmt" "fmt"
"git.cin.su/m/Assistant/adapters/models" "git.cin.su/m/Assistant/adapters/models"
"git.cin.su/m/Assistant/adapters/multimodal"
"git.cin.su/m/Assistant/contracts/capability" "git.cin.su/m/Assistant/contracts/capability"
"git.cin.su/m/Assistant/contracts/cognitive" "git.cin.su/m/Assistant/contracts/cognitive"
"git.cin.su/m/Assistant/contracts/identity" "git.cin.su/m/Assistant/contracts/identity"
@@ -170,6 +171,26 @@ func main() {
mux.HandleFunc("POST /v1/advisories", gateway.advisoryHandler) mux.HandleFunc("POST /v1/advisories", gateway.advisoryHandler)
mux.HandleFunc("POST /v1/interpretations", gateway.interpretationHandler) mux.HandleFunc("POST /v1/interpretations", gateway.interpretationHandler)
mux.HandleFunc("POST /v1/task-evaluations/execute", gateway.taskEvaluationHandler) mux.HandleFunc("POST /v1/task-evaluations/execute", gateway.taskEvaluationHandler)
imageGenerator := multimodal.ImageGenerator(multimodal.ImageGenerationClient{BaseURL: os.Getenv("LOCAL_IMAGE_PROVIDER_URL"), Client: &http.Client{Timeout: 300 * time.Second}})
if strings.EqualFold(strings.TrimSpace(os.Getenv("LOCAL_IMAGE_PROVIDER_KIND")), "comfyui") {
imageGenerator = multimodal.ComfyImageGenerationClient{BaseURL: os.Getenv("LOCAL_IMAGE_PROVIDER_URL"), Client: &http.Client{Timeout: 300 * time.Second}, PollTimeout: 15 * time.Minute}
}
multimodalGateway := multimodalGateway{
speech: multimodal.SpeechClient{BaseURL: envOr("LOCAL_SPEECH_PROVIDER_URL", "http://llm-local-speech:8080"), Client: &http.Client{Timeout: 120 * time.Second}},
vision: multimodal.VisionClient{BaseURL: envOr("LOCAL_VISION_PROVIDER_URL", "http://llm-local-vision:8080/v1"), Client: &http.Client{Timeout: 180 * time.Second}},
imageGenerator: imageGenerator,
imageProviderURL: os.Getenv("LOCAL_IMAGE_PROVIDER_URL"),
video: multimodal.VideoUnderstandingClient{BaseURL: os.Getenv("LOCAL_VIDEO_PROVIDER_URL"), Client: &http.Client{Timeout: 300 * time.Second}},
model: envOr("LOCAL_VISION_MODEL", "/models/gemma-4-26B-A4B-it-UD-Q4_K_M.gguf"),
imageModel: os.Getenv("LOCAL_IMAGE_MODEL"),
videoModel: os.Getenv("LOCAL_VIDEO_MODEL"),
now: time.Now,
}
mux.HandleFunc("POST /v1/multimodal/transcribe", multimodalGateway.handler(capability.MultimodalTranscribe))
mux.HandleFunc("POST /v1/multimodal/understand", multimodalGateway.handler(capability.MultimodalUnderstand))
mux.HandleFunc("POST /v1/multimodal/ocr", multimodalGateway.handler(capability.MultimodalRecognizeOCR))
mux.HandleFunc("POST /v1/multimodal/video-understand", multimodalGateway.handler(capability.MultimodalUnderstandVideo))
mux.HandleFunc("POST /v1/multimodal/generate-image", multimodalGateway.handler(capability.MultimodalGenerateImage))
discovery := discoveryService{resolver: resolver, providers: discoveries, owner: identity.PrincipalID(envOr("COGNITIVE_OWNER_PRINCIPAL_ID", "domovoy-creator-principal")), now: time.Now} discovery := discoveryService{resolver: resolver, providers: discoveries, owner: identity.PrincipalID(envOr("COGNITIVE_OWNER_PRINCIPAL_ID", "domovoy-creator-principal")), now: time.Now}
mux.HandleFunc("POST /v1/provider-discovery/{provider}", discovery.handler) mux.HandleFunc("POST /v1/provider-discovery/{provider}", discovery.handler)
mux.Handle("POST /v1/provider-discovery/{provider}/qualify", modelQualificationHandler(discoveries, string(envOr("COGNITIVE_OWNER_PRINCIPAL_ID", "domovoy-creator-principal")), httpQualificationObserver{resolver: resolver})) mux.Handle("POST /v1/provider-discovery/{provider}/qualify", modelQualificationHandler(discoveries, string(envOr("COGNITIVE_OWNER_PRINCIPAL_ID", "domovoy-creator-principal")), httpQualificationObserver{resolver: resolver}))
+102
View File
@@ -0,0 +1,102 @@
package main
import (
"encoding/json"
"io"
"log"
"net/http"
"strings"
"time"
"git.cin.su/m/Assistant/adapters/multimodal"
"git.cin.su/m/Assistant/contracts/capability"
)
const maxMultimodalUploadBytes = 12 << 20
type multimodalGateway struct {
speech multimodal.SpeechClient
vision multimodal.VisionClient
imageGenerator multimodal.ImageGenerator
imageProviderURL string
video multimodal.VideoUnderstandingClient
model string
imageModel string
videoModel string
now func() time.Time
}
func (g multimodalGateway) handler(operation capability.MultimodalOperation) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
r.Body = http.MaxBytesReader(w, r.Body, maxMultimodalUploadBytes)
if err := r.ParseMultipartForm(maxMultimodalUploadBytes); err != nil {
http.Error(w, "media upload is too large or invalid", http.StatusRequestEntityTooLarge)
return
}
if operation == capability.MultimodalGenerateImage {
if strings.TrimSpace(g.imageProviderURL) == "" {
http.Error(w, "local image generation capability unavailable", http.StatusBadGateway)
return
}
image, err := g.imageGenerator.Generate(r.Context(), g.imageModel, r.FormValue("prompt"), r.FormValue("size"))
if err != nil || strings.TrimSpace(image.DataBase64) == "" {
logMultimodalFailure(operation, err)
http.Error(w, "local image generation capability unavailable", http.StatusBadGateway)
return
}
result := capability.MultimodalResponse{Operation: operation, Provider: "local-image-generator", ObservedAt: g.now().UnixMilli(), ContentType: image.ContentType, DataBase64: image.DataBase64}
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(result)
return
}
file, header, err := r.FormFile("file")
if err != nil {
http.Error(w, "media file is required", http.StatusBadRequest)
return
}
defer file.Close()
data, err := io.ReadAll(io.LimitReader(file, maxMultimodalUploadBytes+1))
if err != nil || len(data) == 0 || len(data) > maxMultimodalUploadBytes {
http.Error(w, "media file is invalid", http.StatusBadRequest)
return
}
contentType := header.Header.Get("Content-Type")
if contentType == "" {
contentType = "application/octet-stream"
}
var text string
if operation == capability.MultimodalTranscribe {
text, err = g.speech.Transcribe(r.Context(), header.Filename, contentType, data)
} else if operation == capability.MultimodalUnderstandVideo {
if strings.TrimSpace(g.video.BaseURL) == "" {
http.Error(w, "local video understanding capability unavailable", http.StatusBadGateway)
return
}
text, err = g.video.Describe(r.Context(), g.videoModel, header.Filename, contentType, data, r.FormValue("instruction"))
} else {
instruction := "Опиши, что изображено на фотографии, коротко и по существу. Не выдумывай нечитаемые детали."
if operation == capability.MultimodalRecognizeOCR {
instruction = "Точно распознай весь читаемый текст на изображении. Сохрани строки и символы; нечитаемые места обозначь как [неразборчиво]. Не добавляй пояснений."
}
text, err = g.vision.Describe(r.Context(), g.model, contentType, data, instruction)
}
if err != nil {
logMultimodalFailure(operation, err)
http.Error(w, "local media capability unavailable", http.StatusBadGateway)
return
}
result := capability.MultimodalResponse{Operation: operation, Text: strings.TrimSpace(text), ObservedAt: g.now().UnixMilli(), ConfidenceNote: "Результат получен локальным специализированным сервисом; нечитаемые или неоднозначные фрагменты требуют проверки."}
if operation == capability.MultimodalTranscribe {
result.Provider = "local-whisper"
} else {
result.Provider = "local-vision"
}
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(result)
}
}
func logMultimodalFailure(operation capability.MultimodalOperation, err error) {
// Keep provider details in server diagnostics; never return them to chat.
log.Printf("multimodal operation %s failed: %v", operation, err)
}
+75
View File
@@ -0,0 +1,75 @@
package main
import (
"bytes"
"context"
"encoding/json"
"mime/multipart"
"net/http"
"net/http/httptest"
"testing"
"time"
"git.cin.su/m/Assistant/adapters/multimodal"
"git.cin.su/m/Assistant/contracts/capability"
)
func TestMultimodalTranscriptionHandlerReturnsLocalResult(t *testing.T) {
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/inference" || r.Method != http.MethodPost {
t.Fatalf("unexpected upstream request: %s %s", r.Method, r.URL.Path)
}
_ = json.NewEncoder(w).Encode(map[string]string{"text": "добрый день"})
}))
defer upstream.Close()
server := multimodalGateway{speech: multimodal.SpeechClient{BaseURL: upstream.URL, Client: upstream.Client()}, now: time.Now}
request := multipartRequest(t, "/v1/multimodal/transcribe", "voice.wav", []byte("audio"))
response := httptest.NewRecorder()
server.handler(capability.MultimodalTranscribe)(response, request)
if response.Code != http.StatusOK || !bytes.Contains(response.Body.Bytes(), []byte("добрый день")) {
t.Fatalf("status=%d body=%s", response.Code, response.Body.String())
}
}
func TestMultimodalImageGenerationFailsClosedWhenSkillIsUnbound(t *testing.T) {
server := multimodalGateway{now: time.Now}
body := &bytes.Buffer{}
writer := multipart.NewWriter(body)
if err := writer.WriteField("prompt", "green bird"); err != nil || writer.Close() != nil {
t.Fatal("failed to build generation request")
}
request := httptest.NewRequest(http.MethodPost, "/v1/multimodal/generate-image", body)
request.Header.Set("Content-Type", writer.FormDataContentType())
response := httptest.NewRecorder()
server.handler(capability.MultimodalGenerateImage)(response, request)
if response.Code != http.StatusBadGateway || !bytes.Contains(response.Body.Bytes(), []byte("unavailable")) {
t.Fatalf("status=%d body=%s", response.Code, response.Body.String())
}
}
func TestMultimodalVideoUnderstandingFailsClosedWhenSkillIsUnbound(t *testing.T) {
server := multimodalGateway{now: time.Now}
request := multipartRequest(t, "/v1/multimodal/video-understand", "clip.mp4", []byte("video"))
response := httptest.NewRecorder()
server.handler(capability.MultimodalUnderstandVideo)(response, request)
if response.Code != http.StatusBadGateway || !bytes.Contains(response.Body.Bytes(), []byte("unavailable")) {
t.Fatalf("status=%d body=%s", response.Code, response.Body.String())
}
}
func multipartRequest(t *testing.T, path, filename string, data []byte) *http.Request {
t.Helper()
body := &bytes.Buffer{}
writer := multipart.NewWriter(body)
part, err := writer.CreateFormFile("file", filename)
if err != nil {
t.Fatal(err)
}
if _, err := part.Write(data); err != nil || writer.Close() != nil {
t.Fatal("failed to build multipart request")
}
request := httptest.NewRequest(http.MethodPost, path, body)
request = request.WithContext(context.Background())
request.Header.Set("Content-Type", writer.FormDataContentType())
return request
}
+3
View File
@@ -12,6 +12,7 @@ type chatRouteKind string
const ( const (
chatRouteResearch chatRouteKind = "RESEARCH" chatRouteResearch chatRouteKind = "RESEARCH"
chatRouteConversation chatRouteKind = "CONVERSATION" chatRouteConversation chatRouteKind = "CONVERSATION"
chatRouteMedia chatRouteKind = "MEDIA"
chatRouteAction chatRouteKind = "ACTION" chatRouteAction chatRouteKind = "ACTION"
chatRouteAgent chatRouteKind = "AGENT" chatRouteAgent chatRouteKind = "AGENT"
) )
@@ -42,6 +43,8 @@ func buildChatRoutePlan(result interactiondomain.SemanticResult) []chatRoute {
switch capability.CapabilityKind(hint) { switch capability.CapabilityKind(hint) {
case capability.CapabilityKindWebResearch: case capability.CapabilityKindWebResearch:
add(chatRouteResearch, capability.CapabilityKindWebResearch, "truth-research-v1", 10) add(chatRouteResearch, capability.CapabilityKindWebResearch, "truth-research-v1", 10)
case capability.CapabilityKindImageGeneration:
add(chatRouteMedia, capability.CapabilityKindImageGeneration, "local-image-generation-v1", 5)
case capability.CapabilityKindCognitiveReasoning, capability.CapabilityKindVerifyFact: case capability.CapabilityKindCognitiveReasoning, capability.CapabilityKindVerifyFact:
add(chatRouteConversation, capability.CapabilityKind(hint), "local-reasoning-v1", 20) add(chatRouteConversation, capability.CapabilityKind(hint), "local-reasoning-v1", 20)
case capability.CapabilityKindScheduleReminder: case capability.CapabilityKindScheduleReminder:
+47 -1
View File
@@ -6,6 +6,8 @@ import (
"encoding/json" "encoding/json"
"fmt" "fmt"
"hash/fnv" "hash/fnv"
"io"
"mime/multipart"
"net/http" "net/http"
"net/url" "net/url"
"strings" "strings"
@@ -23,6 +25,7 @@ import (
type cognitiveInterpretationClient struct { type cognitiveInterpretationClient struct {
baseURL *url.URL baseURL *url.URL
client *http.Client client *http.Client
media *http.Client
} }
// ReviewResponse asks the cognitive gateway to turn a locally assembled draft // ReviewResponse asks the cognitive gateway to turn a locally assembled draft
@@ -83,7 +86,7 @@ func newCognitiveInterpretationClient(rawURL string, client *http.Client) (cogni
if client == nil { if client == nil {
client = http.DefaultClient client = http.DefaultClient
} }
return cognitiveInterpretationClient{baseURL: base, client: client}, nil return cognitiveInterpretationClient{baseURL: base, client: client, media: &http.Client{Timeout: 15 * time.Minute}}, nil
} }
func (c cognitiveInterpretationClient) Interpret(ctx context.Context, request cognitive.GatewayInterpretationRequest) (cognitive.GatewayInterpretationResponse, error) { func (c cognitiveInterpretationClient) Interpret(ctx context.Context, request cognitive.GatewayInterpretationRequest) (cognitive.GatewayInterpretationResponse, error) {
@@ -113,6 +116,49 @@ func (c cognitiveInterpretationClient) Interpret(ctx context.Context, request co
return result, nil return result, nil
} }
func (c cognitiveInterpretationClient) GenerateImage(ctx context.Context, prompt, size string) (capability.MultimodalResponse, error) {
var body bytes.Buffer
writer := multipart.NewWriter(&body)
if err := writer.WriteField("prompt", prompt); err != nil {
return capability.MultimodalResponse{}, err
}
if strings.TrimSpace(size) != "" {
if err := writer.WriteField("size", size); err != nil {
return capability.MultimodalResponse{}, err
}
}
if err := writer.Close(); err != nil {
return capability.MultimodalResponse{}, err
}
target := *c.baseURL
target.Path = "/v1/multimodal/generate-image"
request, err := http.NewRequestWithContext(ctx, http.MethodPost, target.String(), &body)
if err != nil {
return capability.MultimodalResponse{}, err
}
request.Header.Set("Content-Type", writer.FormDataContentType())
client := c.media
if client == nil {
client = c.client
}
response, err := client.Do(request)
if err != nil {
return capability.MultimodalResponse{}, err
}
defer response.Body.Close()
if response.StatusCode != http.StatusOK {
return capability.MultimodalResponse{}, fmt.Errorf("image generation status %d", response.StatusCode)
}
var result capability.MultimodalResponse
if err := json.NewDecoder(io.LimitReader(response.Body, 32<<20)).Decode(&result); err != nil {
return capability.MultimodalResponse{}, err
}
if strings.TrimSpace(result.DataBase64) == "" {
return capability.MultimodalResponse{}, fmt.Errorf("image generation returned empty payload")
}
return result, nil
}
func shouldRequestBroaderInterpretation(result interactiondomain.SemanticResult) bool { func shouldRequestBroaderInterpretation(result interactiondomain.SemanticResult) bool {
return shouldRequestSemanticInterpretation(result) return shouldRequestSemanticInterpretation(result)
} }
@@ -5,9 +5,11 @@ import (
"encoding/json" "encoding/json"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"strings"
"testing" "testing"
"time" "time"
"git.cin.su/m/Assistant/contracts/capability"
"git.cin.su/m/Assistant/contracts/cognitive" "git.cin.su/m/Assistant/contracts/cognitive"
"git.cin.su/m/Assistant/contracts/identity" "git.cin.su/m/Assistant/contracts/identity"
"git.cin.su/m/Assistant/contracts/interaction" "git.cin.su/m/Assistant/contracts/interaction"
@@ -40,6 +42,30 @@ func TestReviewResponseUsesCognitiveGatewayAndReturnsOnlyDraftAnswer(t *testing.
} }
} }
func TestGenerateImageUsesDedicatedCognitiveMediaRoute(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost || r.URL.Path != "/v1/multimodal/generate-image" {
t.Fatalf("unexpected image route: %s %s", r.Method, r.URL.Path)
}
if err := r.ParseMultipartForm(1 << 20); err != nil {
t.Fatalf("parse image request: %v", err)
}
if !strings.Contains(r.FormValue("prompt"), "собаку") {
t.Fatalf("prompt was not preserved: %q", r.FormValue("prompt"))
}
_ = json.NewEncoder(w).Encode(capability.MultimodalResponse{Operation: capability.MultimodalGenerateImage, ContentType: "image/png", DataBase64: "cG5n"})
}))
defer server.Close()
client, err := newCognitiveInterpretationClient(server.URL, server.Client())
if err != nil {
t.Fatal(err)
}
image, err := client.GenerateImage(context.Background(), "нарисуй собаку", "")
if err != nil || image.DataBase64 != "cG5n" || image.ContentType != "image/png" {
t.Fatalf("image=%+v err=%v", image, err)
}
}
func TestReviewPreservesResearchEvidence(t *testing.T) { func TestReviewPreservesResearchEvidence(t *testing.T) {
draft := "Нашёл вариант. Источник: Aviasales (https://www.aviasales.ru/search)" draft := "Нашёл вариант. Источник: Aviasales (https://www.aviasales.ru/search)"
if reviewPreservesResearchEvidence("Нашёл вариант на Авиасейлс.", draft) { if reviewPreservesResearchEvidence("Нашёл вариант на Авиасейлс.", draft) {
+123
View File
@@ -0,0 +1,123 @@
package main
import (
"testing"
"time"
"git.cin.su/m/Assistant/contracts/capability"
interactiondomain "git.cin.su/m/Assistant/modules/interaction/domain"
)
func TestHouseholdResearchAndActionRoutesRemainIndependent(t *testing.T) {
candidate := interactiondomain.BuildDeterministicCandidate("проверь погоду в Казани, найди кафе рядом и напомни вечером позвонить", "", time.Unix(10, 0).UnixMilli())
planned := buildChatRoutePlan(interactiondomain.SemanticResult{NeedsCognition: candidate.NeedsCognition, CapabilityHints: candidate.CapabilityHints})
want := map[chatRouteKind]bool{chatRouteResearch: false, chatRouteConversation: false, chatRouteAction: false}
for _, route := range planned {
if _, ok := want[route.Kind]; ok {
want[route.Kind] = true
}
}
for kind, present := range want {
if !present {
t.Fatalf("route %s missing for mixed household request: candidate=%+v routes=%+v", kind, candidate, planned)
}
}
}
func TestHouseholdCreativeRouteDoesNotUseResearch(t *testing.T) {
candidate := interactiondomain.BuildDeterministicCandidate("расскажи сказку о Колобке", "", time.Unix(10, 0).UnixMilli())
for _, hint := range candidate.CapabilityHints {
if capability.CapabilityKind(hint) == capability.CapabilityKindWebResearch {
t.Fatalf("creative request selected web research: %+v", candidate)
}
}
if !interactiondomain.IsCreativeRequest("расскажи сказку о Колобке") {
t.Fatal("creative request was not recognized")
}
}
func TestHouseholdMarketplaceBrandSearchUsesResearchRoute(t *testing.T) {
candidate := interactiondomain.BuildDeterministicCandidate("найди на WB бренд marka-wear", "", time.Unix(10, 0).UnixMilli())
decomposition := interactiondomain.DecomposeTaskWithLexicon("найди на WB бренд marka-wear", interactiondomain.TaskLexicon{ObjectAliases: []interactiondomain.TaskLexiconEntry{{Alias: "бренд", Value: "BRAND"}}, SourceAliases: []interactiondomain.TaskLexiconEntry{{Alias: "wb", Value: "WILDBERRIES"}}})
if decomposition.Intent != interactiondomain.TaskIntentSearch || decomposition.Object != "BRAND" || decomposition.Source != "WILDBERRIES" || decomposition.Target != "marka-wear" || len(decomposition.Missing) != 0 {
t.Fatalf("marketplace request was not split into executable atoms: %+v", decomposition)
}
routes := buildChatRoutePlan(interactiondomain.SemanticResult{NeedsCognition: candidate.NeedsCognition, CapabilityHints: candidate.CapabilityHints})
if !hasChatRoute(routes, chatRouteResearch) {
t.Fatalf("marketplace brand request did not select research route: candidate=%+v routes=%+v", candidate, routes)
}
}
func TestHouseholdSemanticMemoryControlsMarketplaceAlias(t *testing.T) {
withoutMemory := interactiondomain.DecomposeTask("найди на WB бренд marka-wear")
if withoutMemory.Source != "" || withoutMemory.Object != "" {
t.Fatalf("core invented marketplace meaning without memory: %+v", withoutMemory)
}
withMemory := interactiondomain.DecomposeTaskWithLexicon("поищи марку marka-wear в Wildberries", interactiondomain.TaskLexicon{
ObjectAliases: []interactiondomain.TaskLexiconEntry{{Alias: "марку", Value: "BRAND"}},
SourceAliases: []interactiondomain.TaskLexiconEntry{{Alias: "wildberries", Value: "WILDBERRIES"}},
})
if withMemory.Intent != interactiondomain.TaskIntentSearch || withMemory.Object != "BRAND" || withMemory.Source != "WILDBERRIES" || withMemory.Target != "marka-wear" || !withMemory.NeedsResearch {
t.Fatalf("semantic memory did not control marketplace decomposition: %+v", withMemory)
}
}
func TestHouseholdBearDrawingUsesCreativeLocalRoute(t *testing.T) {
candidate := interactiondomain.BuildDeterministicCandidate("нарисуй бурого медведя", "", time.Unix(10, 0).UnixMilli())
decomposition := interactiondomain.DecomposeTaskWithLexicon("нарисуй бурого медведя", interactiondomain.TaskLexicon{ObjectAliases: []interactiondomain.TaskLexiconEntry{{Alias: "медвед", Value: "IMAGE"}}})
if decomposition.Intent != interactiondomain.TaskIntentCreate || decomposition.Object != "IMAGE" || decomposition.Target != "бурого медведя" || decomposition.NeedsResearch {
t.Fatalf("creative request was not split into local execution atoms: %+v", decomposition)
}
routes := buildChatRoutePlan(interactiondomain.SemanticResult{NeedsCognition: candidate.NeedsCognition, CapabilityHints: candidate.CapabilityHints})
if !hasChatRoute(routes, chatRouteMedia) {
t.Fatalf("creative image request did not select media route: candidate=%+v routes=%+v", candidate, routes)
}
for _, hint := range candidate.CapabilityHints {
if capability.CapabilityKind(hint) == capability.CapabilityKindWebResearch {
t.Fatalf("bear drawing selected web research: %+v", candidate)
}
}
for _, message := range []string{"нарисуй бурого медведя", "нарисуй птицу", "нарисуй зеленую птицу", "нарисуй птицу в HD", "нарисуй коня"} {
if got := interactiondomain.CreativeDeterministicReply(message); got != "" {
t.Fatalf("creative request %q received a canned reply=%q", message, got)
}
}
}
func TestHouseholdRouteMatrixKeepsMeaningBoundaries(t *testing.T) {
tests := []struct {
message string
wantResearch bool
wantCreative bool
wantHint string
}{
{message: "найди на WB бренд marka-wear", wantResearch: true},
{message: "какая погода в Казани?", wantResearch: true},
{message: "найди билеты до Сочи", wantResearch: true},
{message: "нарисуй птицу", wantCreative: true},
{message: "расскажи сказку о Колобке", wantCreative: true},
{message: "что такое гравитация?", wantHint: "COGNITIVE_REASONING"},
}
for _, test := range tests {
candidate := interactiondomain.BuildDeterministicCandidate(test.message, "", time.Unix(10, 0).UnixMilli())
if interactiondomain.IsCreativeRequest(test.message) != test.wantCreative {
t.Fatalf("message=%q creative=%v want=%v candidate=%+v", test.message, interactiondomain.IsCreativeRequest(test.message), test.wantCreative, candidate)
}
gotResearch := false
gotHint := false
for _, hint := range candidate.CapabilityHints {
if hint == "WEB_RESEARCH" {
gotResearch = true
}
if hint == test.wantHint {
gotHint = true
}
}
if gotResearch != test.wantResearch {
t.Fatalf("message=%q research=%v want=%v candidate=%+v", test.message, gotResearch, test.wantResearch, candidate)
}
if test.wantHint != "" && !gotHint {
t.Fatalf("message=%q missing hint=%q candidate=%+v", test.message, test.wantHint, candidate)
}
}
}
@@ -0,0 +1,25 @@
package main
import (
"testing"
"git.cin.su/m/Assistant/contracts/interaction"
)
func TestSpecializedInformationReplyIsNotOverwrittenByResearch(t *testing.T) {
for _, kind := range []interaction.InformationRequestKind{
interaction.InformationRequestMemoryInventory,
interaction.InformationRequestTimeDifference,
interaction.InformationRequestCapabilityInventory,
interaction.InformationRequestCognitiveModelInventory,
interaction.InformationRequestCognitiveChannelStatus,
interaction.InformationRequestCognitiveRecovery,
} {
if shouldRunGeneralResearch(kind) {
t.Fatalf("specialized information request %q must bypass general research", kind)
}
}
if !shouldRunGeneralResearch("") {
t.Fatal("ordinary research requests must still use general research")
}
}
+115 -30
View File
@@ -510,6 +510,7 @@ func main() {
return return
} }
var effectiveTaskMessage = interactiondomain.StripAssistantVocative(utterance.Content) var effectiveTaskMessage = interactiondomain.StripAssistantVocative(utterance.Content)
creativeUserRequest := interactiondomain.IsCreativeRequest(utterance.Content)
continuationMessage := "" continuationMessage := ""
travelContinuation := false travelContinuation := false
var previousTask *interaction.HumanUtterance var previousTask *interaction.HumanUtterance
@@ -531,13 +532,15 @@ func main() {
log.Printf("interaction principal history loaded: %d utterances", len(recentUserHistory)) log.Printf("interaction principal history loaded: %d utterances", len(recentUserHistory))
} }
conversationHistory = append(conversationHistory, recentUserHistory...) conversationHistory = append(conversationHistory, recentUserHistory...)
if continuation := interactiondomain.ContinueLocalSearch(effectiveTaskMessage, conversationHistory); continuation != "" { if !creativeUserRequest {
continuationMessage = continuation if continuation := interactiondomain.ContinueLocalSearch(effectiveTaskMessage, conversationHistory); continuation != "" {
effectiveTaskMessage = continuation continuationMessage = continuation
} else if continuation := interactiondomain.ContinueTravelOrigin(effectiveTaskMessage, conversationHistory); continuation != "" { effectiveTaskMessage = continuation
continuationMessage = continuation } else if continuation := interactiondomain.ContinueTravelOrigin(effectiveTaskMessage, conversationHistory); continuation != "" {
travelContinuation = true continuationMessage = continuation
effectiveTaskMessage = continuation travelContinuation = true
effectiveTaskMessage = continuation
}
} }
if request.ConversationID != "" && interactiondomain.IsRetryRequest(utterance.Content) { if request.ConversationID != "" && interactiondomain.IsRetryRequest(utterance.Content) {
if len(conversationHistory) > 0 { if len(conversationHistory) > 0 {
@@ -567,12 +570,12 @@ func main() {
semantic = continuedSemantic semantic = continuedSemantic
} }
} }
if travelContinuation && interactiondomain.TravelSearchClarification(effectiveTaskMessage, conversationHistory) == "" { if !creativeUserRequest && travelContinuation && interactiondomain.TravelSearchClarification(effectiveTaskMessage, conversationHistory) == "" {
semantic.CapabilityHints = append(semantic.CapabilityHints, string(capability.CapabilityKindWebResearch)) semantic.CapabilityHints = append(semantic.CapabilityHints, string(capability.CapabilityKindWebResearch))
semantic.NeedsCognition = false semantic.NeedsCognition = false
semantic.Interpretation.NeedsCognition = false semantic.Interpretation.NeedsCognition = false
} }
creativeRequest := interactiondomain.IsCreativeRequest(utterance.Content) || interactiondomain.IsCreativeRequest(effectiveTaskMessage) creativeRequest := creativeUserRequest || interactiondomain.IsCreativeRequest(effectiveTaskMessage)
creativeReply := interactiondomain.CreativeDeterministicReply(utterance.Content) creativeReply := interactiondomain.CreativeDeterministicReply(utterance.Content)
if creativeReply == "" { if creativeReply == "" {
creativeReply = interactiondomain.CreativeDeterministicReply(effectiveTaskMessage) creativeReply = interactiondomain.CreativeDeterministicReply(effectiveTaskMessage)
@@ -591,6 +594,9 @@ func main() {
semantic.NeedsCognition = true semantic.NeedsCognition = true
semantic.Interpretation.NeedsCognition = true semantic.Interpretation.NeedsCognition = true
semantic.CapabilityHints = append(semantic.CapabilityHints, string(capability.CapabilityKindCognitiveReasoning)) semantic.CapabilityHints = append(semantic.CapabilityHints, string(capability.CapabilityKindCognitiveReasoning))
if creativeFallbackReply != "" {
semantic.Reply = creativeFallbackReply
}
} }
} }
locationReply := interactiondomain.LocationContextReply(utterance.Content) locationReply := interactiondomain.LocationContextReply(utterance.Content)
@@ -626,6 +632,13 @@ func main() {
cognitiveAvailable := true cognitiveAvailable := true
semanticInterpretationRequested := shouldRequestSemanticInterpretation(semantic) semanticInterpretationRequested := shouldRequestSemanticInterpretation(semantic)
semanticReflexAudit := shouldAuditDeterministicReflex(semantic, string(utterance.ID), now) semanticReflexAudit := shouldAuditDeterministicReflex(semantic, string(utterance.ID), now)
// A selected specialized media route already has an executable semantic
// outcome. Do not block it on the general cognitive channel: image
// generation must reach its own adapter even when cognition is degraded.
if hasChatRoute(routePlan, chatRouteMedia) {
semanticInterpretationRequested = false
semanticReflexAudit = false
}
if identityReply != "" { if identityReply != "" {
semanticReflexAudit = false semanticReflexAudit = false
} }
@@ -649,13 +662,15 @@ func main() {
cognitiveAvailable = false cognitiveAvailable = false
} }
} }
if travelReply := interactiondomain.TravelSearchClarification(effectiveTaskMessage, conversationHistory); travelReply != "" { if !creativeRequest {
semantic.Interpretation.Clarification = interaction.ClarificationRequired if travelReply := interactiondomain.TravelSearchClarification(effectiveTaskMessage, conversationHistory); travelReply != "" {
semantic.Interpretation.Applicability = interaction.SemanticCandidateClarify semantic.Interpretation.Clarification = interaction.ClarificationRequired
semantic.Reply = travelReply semantic.Interpretation.Applicability = interaction.SemanticCandidateClarify
deterministicReply = travelReply semantic.Reply = travelReply
semantic.NeedsCognition = false deterministicReply = travelReply
semantic.Interpretation.NeedsCognition = false semantic.NeedsCognition = false
semantic.Interpretation.NeedsCognition = false
}
} }
if err := interactionRepository.AppendUtterance(r.Context(), conversation, utterance, intent, relations); err != nil { if err := interactionRepository.AppendUtterance(r.Context(), conversation, utterance, intent, relations); err != nil {
http.Error(w, "save chat message", http.StatusInternalServerError) http.Error(w, "save chat message", http.StatusInternalServerError)
@@ -754,7 +769,7 @@ func main() {
needsCognition = false needsCognition = false
} }
researchBacked := false researchBacked := false
if hasChatRoute(routePlan, chatRouteResearch) && semantic.Interpretation.Clarification != interaction.ClarificationRequired { if hasChatRoute(routePlan, chatRouteResearch) && semantic.Interpretation.Clarification != interaction.ClarificationRequired && shouldRunGeneralResearch(semantic.InformationRequestKind) {
if planID, idErr := ids.New(); idErr == nil { if planID, idErr := ids.New(); idErr == nil {
parts := interactiondomain.IndependentRequestParts(effectiveTaskMessage) parts := interactiondomain.IndependentRequestParts(effectiveTaskMessage)
replies := make([]string, 0, len(parts)) replies := make([]string, 0, len(parts))
@@ -763,10 +778,11 @@ func main() {
replies = append(replies, "По запросу «"+part+"»: "+clarification) replies = append(replies, "По запросу «"+part+"»: "+clarification)
continue continue
} }
searchQuery := interactiondomain.TravelSearchQuery(part, now)
researchCtx, cancelResearch := context.WithTimeout(r.Context(), 8*time.Second) researchCtx, cancelResearch := context.WithTimeout(r.Context(), 8*time.Second)
result, researchErr := researchClient.SearchWeb(researchCtx, epistemics.WebSearchRequest{ result, researchErr := researchClient.SearchWeb(researchCtx, epistemics.WebSearchRequest{
Plan: epistemics.ResearchPlan{ID: epistemics.ResearchPlanID(fmt.Sprintf("plan:%s:%d", planID, index)), Question: part, SourceClasses: []string{"web-search"}, FreshnessPolicy: "current"}, Plan: epistemics.ResearchPlan{ID: epistemics.ResearchPlanID(fmt.Sprintf("plan:%s:%d", planID, index)), Question: searchQuery, SourceClasses: []string{"web-search"}, FreshnessPolicy: "current"},
PropositionKey: fmt.Sprintf("question:%s:%d", utterance.ID, index), Query: part, PropositionKey: fmt.Sprintf("question:%s:%d", utterance.ID, index), Query: searchQuery,
}) })
cancelResearch() cancelResearch()
reply, backed := renderResearchReply(part, result, researchErr) reply, backed := renderResearchReply(part, result, researchErr)
@@ -791,9 +807,11 @@ func main() {
planningReply = helpPath.Reply planningReply = helpPath.Reply
needsCognition = false needsCognition = false
} }
if travelReply := interactiondomain.TravelSearchClarification(effectiveTaskMessage, conversationHistory); travelReply != "" { if !creativeRequest {
planningReply = travelReply if travelReply := interactiondomain.TravelSearchClarification(effectiveTaskMessage, conversationHistory); travelReply != "" {
needsCognition = false planningReply = travelReply
needsCognition = false
}
} }
if planningReply == "" && needsCognition && cognitiveAttempted && !cognitiveAvailable { if planningReply == "" && needsCognition && cognitiveAttempted && !cognitiveAvailable {
planningReply = humanCognitiveDegradedReply(utterance.Content) planningReply = humanCognitiveDegradedReply(utterance.Content)
@@ -809,6 +827,25 @@ func main() {
var delegationID work.DelegationID var delegationID work.DelegationID
var delegationState work.DelegationState var delegationState work.DelegationState
var reminder *continuity.ProspectiveItem var reminder *continuity.ProspectiveItem
var generatedImage capability.MultimodalResponse
mediaHandled := false
mediaAttempted := false
if hasChatRoute(routePlan, chatRouteMedia) && semantic.Interpretation.Clarification != interaction.ClarificationRequired {
mediaAttempted = true
imageCtx, cancelImage := context.WithTimeout(r.Context(), 15*time.Minute)
generatedImage, err = cognitiveClient.GenerateImage(imageCtx, effectiveTaskMessage, "")
cancelImage()
if err == nil && strings.TrimSpace(generatedImage.DataBase64) != "" {
planningReply = "Готово."
semantic.Reply = planningReply
needsCognition = false
mediaHandled = true
} else {
planningReply = "Сейчас не удалось создать изображение."
semantic.Reply = planningReply
needsCognition = false
}
}
if helpPath.Kind == work.HelpPathDirectCapability && len(helpPath.SuggestedCapabilityKinds) > 0 { if helpPath.Kind == work.HelpPathDirectCapability && len(helpPath.SuggestedCapabilityKinds) > 0 {
capabilityKind = helpPath.SuggestedCapabilityKinds[0] capabilityKind = helpPath.SuggestedCapabilityKinds[0]
started := time.Now() started := time.Now()
@@ -893,7 +930,7 @@ func main() {
ConfidenceMillis: semantic.Interpretation.ConfidenceMillis, ConfidenceMillis: semantic.Interpretation.ConfidenceMillis,
}) })
cancelReview() cancelReview()
stableSemanticReply := identityReply != "" || creativeReply != "" || (creativeFallbackReply != "" && !cognitiveAvailable) || semantic.InformationRequestKind != "" || locationReply != "" || deterministicSpeechAct == interaction.SpeechActConversational || deterministicSpeechAct == interaction.SpeechActConfirmation || deterministicSpeechAct == interaction.SpeechActRejection stableSemanticReply := mediaAttempted || identityReply != "" || creativeReply != "" || (creativeFallbackReply != "" && !cognitiveAvailable) || semantic.InformationRequestKind != "" || locationReply != "" || deterministicSpeechAct == interaction.SpeechActConversational || deterministicSpeechAct == interaction.SpeechActConfirmation || deterministicSpeechAct == interaction.SpeechActRejection
if reviewErr != nil { if reviewErr != nil {
log.Printf("cognitive response review unavailable: %v", reviewErr) log.Printf("cognitive response review unavailable: %v", reviewErr)
} else if !stableSemanticReply && (!researchBacked || reviewPreservesResearchEvidence(reviewedReply, planningReply)) { } else if !stableSemanticReply && (!researchBacked || reviewPreservesResearchEvidence(reviewedReply, planningReply)) {
@@ -902,9 +939,12 @@ func main() {
log.Printf("cognitive response review proposed stable semantic rewrite; preserving semantic reply") log.Printf("cognitive response review proposed stable semantic rewrite; preserving semantic reply")
} }
} }
if identityReply == "" { if !mediaAttempted && !mediaHandled && identityReply == "" && creativeReply == "" && creativeFallbackReply == "" {
stableAct := interactiondomain.ClassifySpeechAct(utterance.Content, utterance.CorrectsUtteranceID) stableAct := interactiondomain.ClassifySpeechAct(utterance.Content, utterance.CorrectsUtteranceID)
travelClarification := interactiondomain.TravelSearchClarification(effectiveTaskMessage, conversationHistory) travelClarification := ""
if !creativeRequest {
travelClarification = interactiondomain.TravelSearchClarification(effectiveTaskMessage, conversationHistory)
}
if stableReply, stable := interactiondomain.StableConversationalReply(utterance.Content, stableAct); stable && travelClarification == "" { if stableReply, stable := interactiondomain.StableConversationalReply(utterance.Content, stableAct); stable && travelClarification == "" {
planningReply = stableReply planningReply = stableReply
needsCognition = false needsCognition = false
@@ -937,6 +977,7 @@ func main() {
InterpretationState: utterance.InterpretationState, InterpretationState: utterance.InterpretationState,
NeedsCognition: needsCognition, NeedsCognition: needsCognition,
FallbackText: semantic.Reply, FallbackText: semantic.Reply,
AssistantName: assistantName,
ActiveProjectName: activeFocusName, ActiveProjectName: activeFocusName,
ProjectProvenance: activeFocusProvenance, ProjectProvenance: activeFocusProvenance,
Clarification: semantic.Interpretation.Clarification, Clarification: semantic.Interpretation.Clarification,
@@ -953,6 +994,8 @@ func main() {
DelegationID: string(delegationID), DelegationID: string(delegationID),
DelegationState: string(delegationState), DelegationState: string(delegationState),
Reminder: reminder, Reminder: reminder,
GeneratedImageContentType: generatedImage.ContentType,
GeneratedImageBase64: generatedImage.DataBase64,
} }
w.Header().Set("Content-Type", "application/json") w.Header().Set("Content-Type", "application/json")
w.Header().Set("Cache-Control", "no-store") w.Header().Set("Cache-Control", "no-store")
@@ -1546,6 +1589,10 @@ func main() {
} }
} }
func shouldRunGeneralResearch(kind interaction.InformationRequestKind) bool {
return kind == ""
}
func creatorSessionHandler(bootstrapToken, creatorUsername, creatorPassword string, sessionKey []byte, now func() time.Time) http.HandlerFunc { func creatorSessionHandler(bootstrapToken, creatorUsername, creatorPassword string, sessionKey []byte, now func() time.Time) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) { return func(w http.ResponseWriter, r *http.Request) {
authenticated := identitydomain.BootstrapMatches(bootstrapToken, bearerToken(r)) authenticated := identitydomain.BootstrapMatches(bootstrapToken, bearerToken(r))
@@ -1670,7 +1717,7 @@ func renderResearchReply(query string, result epistemics.ResearchResult, researc
if finding.State != epistemics.FindingSupported || strings.TrimSpace(finding.Excerpt.Text) == "" || strings.TrimSpace(finding.Source.Locator) == "" { if finding.State != epistemics.FindingSupported || strings.TrimSpace(finding.Excerpt.Text) == "" || strings.TrimSpace(finding.Source.Locator) == "" {
continue continue
} }
excerpt := strings.TrimSpace(finding.Excerpt.Text) excerpt := humanResearchExcerpt(query, finding.Excerpt.Text)
limit := 180 limit := 180
if index > 0 { if index > 0 {
limit = 100 limit = 100
@@ -1686,7 +1733,10 @@ func renderResearchReply(query string, result epistemics.ResearchResult, researc
if len(lines) == 0 { if len(lines) == 0 {
return "Проверяемых данных пока не найдено. Могу повторить поиск или уточнить, что именно проверить.", false return "Проверяемых данных пока не найдено. Могу повторить поиск или уточнить, что именно проверить.", false
} }
reply := strings.Join(lines, "\n") if nearestFlightQuery(query) && !containsFlightTimeEvidence(strings.Join(lines, " ")) {
return "Нашёл расписание по маршруту, но точное время ближайшего вылета в доступных фрагментах не указано. Откройте источник и выберите ближайший рейс:\n\n" + strings.Join(lines, "\n\n"), true
}
reply := "Нашёл проверяемые данные:\n\n" + strings.Join(lines, "\n\n")
lowerQuery := strings.ToLower(query) lowerQuery := strings.ToLower(query)
if (strings.Contains(lowerQuery, "цен") || strings.Contains(lowerQuery, "стоим")) && !containsPriceEvidence(reply) { if (strings.Contains(lowerQuery, "цен") || strings.Contains(lowerQuery, "стоим")) && !containsPriceEvidence(reply) {
reply = "Актуальная цена в найденных фрагментах не указана, поэтому не буду придумывать сумму. Вот страницы, где её можно проверить:\n" + reply reply = "Актуальная цена в найденных фрагментах не указана, поэтому не буду придумывать сумму. Вот страницы, где её можно проверить:\n" + reply
@@ -1694,6 +1744,35 @@ func renderResearchReply(query string, result epistemics.ResearchResult, researc
return reply, true return reply, true
} }
func nearestFlightQuery(query string) bool {
lower := strings.ToLower(query)
return strings.Contains(lower, "ближ") && (strings.Contains(lower, "рейс") || strings.Contains(lower, "самолет") || strings.Contains(lower, "самолёт") || strings.Contains(lower, "вылет"))
}
func containsFlightTimeEvidence(value string) bool {
return regexp.MustCompile(`(?m)\b(?:[01]?\d|2[0-3]):[0-5]\d\b`).MatchString(value)
}
func humanResearchExcerpt(query, raw string) string {
text := strings.Join(strings.Fields(strings.TrimSpace(raw)), " ")
if text == "" {
return text
}
parts := strings.Split(text, " — ")
if len(parts) < 2 {
return text
}
lowerQuery := strings.ToLower(query)
for index := len(parts) - 1; index >= 0; index-- {
candidate := strings.TrimSpace(parts[index])
lower := strings.ToLower(candidate)
if len([]rune(candidate)) >= 24 && (strings.Contains(lower, "сейчас") || strings.Contains(lower, "температур") || strings.Contains(lower, "цена") || strings.Contains(lower, "стоим") || strings.Contains(lower, "рейс") || strings.Contains(lower, "вылет") || strings.Contains(lowerQuery, "найди")) {
return candidate
}
}
return text
}
func containsPriceEvidence(value string) bool { func containsPriceEvidence(value string) bool {
pricePattern := regexp.MustCompile(`(?i)(\d[\d\s.,]{2,})(?:\s*(?:₽|руб\.?|р\.?|usd|\$|€))`) pricePattern := regexp.MustCompile(`(?i)(\d[\d\s.,]{2,})(?:\s*(?:₽|руб\.?|р\.?|usd|\$|€))`)
return pricePattern.MatchString(value) return pricePattern.MatchString(value)
@@ -1721,7 +1800,13 @@ func humanMemoryInventory(history []interaction.HumanUtterance, assistantName st
lower := strings.ToLower(message) lower := strings.ToLower(message)
if (strings.Contains(lower, "нравится") || strings.Contains(lower, "люблю")) && utterance.ReceivedAtUnixMillis >= latestPreferenceAt { if (strings.Contains(lower, "нравится") || strings.Contains(lower, "люблю")) && utterance.ReceivedAtUnixMillis >= latestPreferenceAt {
latestPreferenceAt = utterance.ReceivedAtUnixMillis latestPreferenceAt = utterance.ReceivedAtUnixMillis
latestPreference = message latestPreference = strings.TrimSpace(message)
for _, prefix := range []string{"запомни:", "запомни", "помни:"} {
if strings.HasPrefix(strings.ToLower(latestPreference), prefix) {
latestPreference = strings.TrimSpace(latestPreference[len(prefix):])
break
}
}
} }
} }
if latestPreference != "" { if latestPreference != "" {
@@ -1924,9 +2009,9 @@ func humanCognitiveChannelReply(kind interaction.InformationRequestKind, act int
func humanCognitiveDegradedReply(message string) string { func humanCognitiveDegradedReply(message string) string {
if strings.TrimSpace(message) == "" { if strings.TrimSpace(message) == "" {
return "Сейчас когнитивный канал недоступен; Домовой остаётся на связи." return "Сейчас не удалось обработать запрос."
} }
return "Сейчас когнитивный канал недоступен. Повторю запрос через доступный маршрут, когда он восстановится." return "Не удалось получить проверяемые данные. Ответ не выдумываю."
} }
func humanCapabilityList(kinds []capability.CapabilityKind) string { func humanCapabilityList(kinds []capability.CapabilityKind) string {
+20 -1
View File
@@ -15,6 +15,25 @@ import (
interactiondomain "git.cin.su/m/Assistant/modules/interaction/domain" interactiondomain "git.cin.su/m/Assistant/modules/interaction/domain"
) )
func TestHumanResearchExcerptRemovesSearchResultHeading(t *testing.T) {
got := humanResearchExcerpt("погода в Санкт-Петербурге", "Погода в Санкт-Петербурге — Прогноз погоды — Сейчас пасмурно, температура +14°")
if got != "Сейчас пасмурно, температура +14°" || strings.Contains(got, "Прогноз погоды") {
t.Fatalf("human research excerpt = %q", got)
}
}
func TestNearestFlightEvidenceDoesNotClaimMissingDepartureTime(t *testing.T) {
if !nearestFlightQuery("ближайший рейс Москва Казань сегодня") {
t.Fatal("nearest flight query was not recognized")
}
if containsFlightTimeEvidence("Казань составляет 1 ч 25 мин. График вылетов доступен онлайн") {
t.Fatal("duration was mistaken for a departure time")
}
if !containsFlightTimeEvidence("Ближайший вылет в 18:40") {
t.Fatal("departure time was not recognized")
}
}
func TestBearerTokenRequiresBearerScheme(t *testing.T) { func TestBearerTokenRequiresBearerScheme(t *testing.T) {
request := httptest.NewRequest("GET", "/", nil) request := httptest.NewRequest("GET", "/", nil)
request.Header.Set("Authorization", "Basic value") request.Header.Set("Authorization", "Basic value")
@@ -73,7 +92,7 @@ func TestCognitiveConfigurationRequestDoesNotPretendToMutate(t *testing.T) {
func TestCognitiveDegradedReplyKeepsDomovoyAlive(t *testing.T) { func TestCognitiveDegradedReplyKeepsDomovoyAlive(t *testing.T) {
reply := humanCognitiveDegradedReply("Помоги настроить локальную модель") reply := humanCognitiveDegradedReply("Помоги настроить локальную модель")
if !strings.Contains(reply, "когнитивный канал") || !strings.Contains(reply, "Повторю") { if strings.Contains(reply, "когнитивный канал") || strings.Contains(reply, "Повторю") || !strings.Contains(reply, "не выдумываю") {
t.Fatalf("unexpected degraded reply: %q", reply) t.Fatalf("unexpected degraded reply: %q", reply)
} }
} }
+10
View File
@@ -40,6 +40,11 @@ func TestChatPageServedWithoutInternalIDs(t *testing.T) {
if !strings.Contains(body, "версия 0123456789ab") || !strings.Contains(body, "обновлено 22.08.2026 03:04 UTC") || !strings.Contains(body, "datetime=\"2026-08-22T03:04:05Z\"") { if !strings.Contains(body, "версия 0123456789ab") || !strings.Contains(body, "обновлено 22.08.2026 03:04 UTC") || !strings.Contains(body, "datetime=\"2026-08-22T03:04:05Z\"") {
t.Fatalf("/chat does not render build metadata: %q", body) t.Fatalf("/chat does not render build metadata: %q", body)
} }
for _, required := range []string{"media-tools", "audio-input", "image-input", "ocr-input"} {
if !strings.Contains(body, required) {
t.Fatalf("/chat is missing multimodal control %q", required)
}
}
for _, path := range []string{"/", "/chat.html"} { for _, path := range []string{"/", "/chat.html"} {
alternate := httptest.NewRecorder() alternate := httptest.NewRecorder()
handler.ServeHTTP(alternate, httptest.NewRequest(http.MethodGet, path, nil)) handler.ServeHTTP(alternate, httptest.NewRequest(http.MethodGet, path, nil))
@@ -60,6 +65,11 @@ func TestChatScriptExplainsSessionAndRuntimeFailures(t *testing.T) {
t.Fatalf("chat script is missing recovery message %q", required) t.Fatalf("chat script is missing recovery message %q", required)
} }
} }
for _, required := range []string{"uploadMedia", "/api/multimodal/transcribe", "/api/multimodal/understand", "/api/multimodal/ocr"} {
if !strings.Contains(script, required) {
t.Fatalf("chat script is missing multimodal route %q", required)
}
}
} }
func TestPublicSurfacesSeparateChatAndConsole(t *testing.T) { func TestPublicSurfacesSeparateChatAndConsole(t *testing.T) {
+1 -1
View File
@@ -20,7 +20,7 @@ func main() {
// A cold local model may need more than one short HTTP round-trip to // A cold local model may need more than one short HTTP round-trip to
// load on CPU. Keep the browser gateway aligned with the cognition // load on CPU. Keep the browser gateway aligned with the cognition
// deadline instead of turning that wait into a false unavailable reply. // deadline instead of turning that wait into a false unavailable reply.
&http.Client{Timeout: 90 * time.Second}, &http.Client{Timeout: 15 * time.Minute},
envOr("COGNITIVE_RUNTIME_URL", "http://cognitive-worker:8083"), envOr("COGNITIVE_RUNTIME_URL", "http://cognitive-worker:8083"),
envOr("AGENT_WORKER_URL", "http://agent-worker:8090"), envOr("AGENT_WORKER_URL", "http://agent-worker:8090"),
) )
+57
View File
@@ -0,0 +1,57 @@
package main
import (
"bytes"
"encoding/json"
"mime/multipart"
"net/http"
"net/http/httptest"
"testing"
"git.cin.su/m/Assistant/contracts/capability"
"git.cin.su/m/Assistant/contracts/identity"
)
func TestMultimodalUploadRequiresSessionAndProxiesLocalResult(t *testing.T) {
core := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/v1/creator/sessions/current" || r.Header.Get("Authorization") != "Bearer session" {
t.Fatalf("unexpected session request: %s %q", r.URL.Path, r.Header.Get("Authorization"))
}
_ = json.NewEncoder(w).Encode(identity.CreatorSession{ActorID: "actor", PrincipalID: "principal"})
}))
defer core.Close()
cognitive := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/v1/multimodal/transcribe" || r.Method != http.MethodPost {
t.Fatalf("unexpected multimodal request: %s %s", r.Method, r.URL.Path)
}
_ = json.NewEncoder(w).Encode(capability.MultimodalResponse{Operation: capability.MultimodalTranscribe, Text: "локальный текст", Provider: "local-whisper", ObservedAt: 1})
}))
defer cognitive.Close()
handler, err := newHandler(core.URL, "http://continuity.invalid", "http://commit.invalid", core.Client(), cognitive.URL)
if err != nil {
t.Fatal(err)
}
denied := httptest.NewRecorder()
handler.ServeHTTP(denied, httptest.NewRequest(http.MethodPost, "/api/multimodal/transcribe", bytes.NewReader(nil)))
if denied.Code != http.StatusUnauthorized {
t.Fatalf("unauthenticated upload status=%d", denied.Code)
}
body := &bytes.Buffer{}
writer := multipart.NewWriter(body)
part, err := writer.CreateFormFile("file", "voice.wav")
if err != nil {
t.Fatal(err)
}
_, _ = part.Write([]byte("audio"))
if err := writer.Close(); err != nil {
t.Fatal(err)
}
request := httptest.NewRequest(http.MethodPost, "/api/multimodal/transcribe", body)
request.Header.Set("Authorization", "Bearer session")
request.Header.Set("Content-Type", writer.FormDataContentType())
response := httptest.NewRecorder()
handler.ServeHTTP(response, request)
if response.Code != http.StatusOK || !bytes.Contains(response.Body.Bytes(), []byte("локальный текст")) {
t.Fatalf("status=%d body=%s", response.Code, response.Body.String())
}
}
+124 -2
View File
@@ -7,7 +7,9 @@ import (
"fmt" "fmt"
"io" "io"
"io/fs" "io/fs"
"mime/multipart"
"net/http" "net/http"
"net/textproto"
"net/url" "net/url"
"os" "os"
"strings" "strings"
@@ -135,6 +137,10 @@ func (g *gateway) ServeHTTP(w http.ResponseWriter, r *http.Request) {
g.handleChatMessage(w, r) g.handleChatMessage(w, r)
return return
} }
if r.Method == http.MethodPost && strings.HasPrefix(r.URL.Path, "/api/multimodal/") {
g.handleMultimodal(w, r)
return
}
if r.Method == http.MethodGet && r.URL.Path == "/api/fleet/readiness" { if r.Method == http.MethodGet && r.URL.Path == "/api/fleet/readiness" {
g.handleFleetReadiness(w, r) g.handleFleetReadiness(w, r)
return return
@@ -214,6 +220,8 @@ type browserChatResponse struct {
DelegationID string `json:"delegation_id"` DelegationID string `json:"delegation_id"`
DelegationState string `json:"delegation_state"` DelegationState string `json:"delegation_state"`
ReminderScheduled bool `json:"reminder_scheduled"` ReminderScheduled bool `json:"reminder_scheduled"`
GeneratedImageContentType string `json:"generated_image_content_type,omitempty"`
GeneratedImageBase64 string `json:"generated_image_base64,omitempty"`
} }
func (g *gateway) handleAdvisoryPolicy(w http.ResponseWriter, r *http.Request) { func (g *gateway) handleAdvisoryPolicy(w http.ResponseWriter, r *http.Request) {
@@ -479,7 +487,7 @@ func (g *gateway) handleChatMessage(w http.ResponseWriter, r *http.Request) {
status, err = g.postTrustedJSON(r, g.cognitive, "/v1/advisories", "", payload, &advisory) status, err = g.postTrustedJSON(r, g.cognitive, "/v1/advisories", "", payload, &advisory)
if err == nil && status == http.StatusOK && validAdvisoryResponse(advisory, trusted) { if err == nil && status == http.StatusOK && validAdvisoryResponse(advisory, trusted) {
cognition = string(advisory.FinishReason) cognition = string(advisory.FinishReason)
if advisory.FinishReason == cognitive.FinishCompleted { if advisory.FinishReason == cognitive.FinishCompleted && (strings.TrimSpace(saved.FallbackText) == "" || !isBoilerplateAdvisory(advisory.AdvisoryText)) {
reply = advisory.AdvisoryText reply = advisory.AdvisoryText
} }
} }
@@ -507,17 +515,33 @@ func (g *gateway) handleChatMessage(w http.ResponseWriter, r *http.Request) {
DelegationID: saved.DelegationID, DelegationID: saved.DelegationID,
DelegationState: saved.DelegationState, DelegationState: saved.DelegationState,
ReminderScheduled: saved.Reminder != nil, ReminderScheduled: saved.Reminder != nil,
GeneratedImageContentType: saved.GeneratedImageContentType,
GeneratedImageBase64: saved.GeneratedImageBase64,
}) })
} }
func isBoilerplateAdvisory(value string) bool {
normalized := strings.ToLower(strings.TrimSpace(value))
switch normalized {
case "я рядом.", "я рядом", "понял.", "понял", "принял.", "принял":
return true
default:
return strings.Contains(normalized, "я — домов") || strings.Contains(normalized, "я - домов") || strings.Contains(normalized, "меня зовут домов")
}
}
func localChatContext(saved interaction.ChatMessageResponse) []cognitive.ContextExcerpt { func localChatContext(saved interaction.ChatMessageResponse) []cognitive.ContextExcerpt {
assistantName := strings.TrimSpace(saved.AssistantName)
if assistantName == "" {
assistantName = "Домовой"
}
candidates := []cognitive.ContextCandidate{{ candidates := []cognitive.ContextCandidate{{
Excerpt: cognitive.ContextExcerpt{ Excerpt: cognitive.ContextExcerpt{
ID: "assistant-identity", ID: "assistant-identity",
SourceKind: "assistant-identity-policy", SourceKind: "assistant-identity-policy",
SourceID: "domovoy", SourceID: "domovoy",
Sensitivity: cognitive.SensitivityPublic, Sensitivity: cognitive.SensitivityPublic,
Text: "Ты — Домовой, локальный помощник пользователя. Отвечай от имени Домового. Не называй себя Qwen, Alibaba Cloud, поставщиком модели или внутренним runtime. Если спрашивают имя, скажи: «Меня зовут Домовой». Твоя задача — понять смысл запроса, найти путь к цели и довести помощь до полезного конечного результата; не ограничивайся пересказом отказа модели.", Text: "Ты — " + assistantName + ", локальный помощник пользователя. Отвечай от имени «" + assistantName + "». Не называй себя Qwen, Alibaba Cloud, поставщиком модели или внутренним runtime. Если спрашивают имя, скажи: «Меня зовут " + assistantName + ".» Твоя задача — понять смысл запроса, найти путь к цели и довести помощь до полезного конечного результата; не ограничивайся пересказом отказа модели.",
}, },
Relevance: 100, Relevance: 100,
}} }}
@@ -650,6 +674,104 @@ func (g *gateway) postTrustedJSON(r *http.Request, upstream *url.URL, path, auth
return response.StatusCode, nil return response.StatusCode, nil
} }
func (g *gateway) handleMultimodal(w http.ResponseWriter, r *http.Request) {
authorization := r.Header.Get("Authorization")
if authorization == "" {
http.Error(w, "creator session denied", http.StatusUnauthorized)
return
}
var session identity.CreatorSession
status, err := g.getTrustedJSON(r, g.core, "/v1/creator/sessions/current", "", authorization, &session)
if err != nil || status != http.StatusOK || session.ActorID == "" || session.PrincipalID == "" {
http.Error(w, "creator session denied", http.StatusUnauthorized)
return
}
const maxMediaBytes = 12 << 20
r.Body = http.MaxBytesReader(w, r.Body, maxMediaBytes)
if err := r.ParseMultipartForm(maxMediaBytes); err != nil {
http.Error(w, "media upload is too large or invalid", http.StatusRequestEntityTooLarge)
return
}
path := ""
switch r.URL.Path {
case "/api/multimodal/transcribe":
path = "/v1/multimodal/transcribe"
case "/api/multimodal/understand":
path = "/v1/multimodal/understand"
case "/api/multimodal/ocr":
path = "/v1/multimodal/ocr"
case "/api/multimodal/video-understand":
path = "/v1/multimodal/video-understand"
case "/api/multimodal/generate-image":
path = "/v1/multimodal/generate-image"
default:
http.Error(w, "multimodal route not found", http.StatusNotFound)
return
}
body := &bytes.Buffer{}
writer := multipart.NewWriter(body)
if r.URL.Path == "/api/multimodal/generate-image" {
_ = writer.WriteField("prompt", r.FormValue("prompt"))
_ = writer.WriteField("size", r.FormValue("size"))
} else {
if r.URL.Path == "/api/multimodal/video-understand" {
_ = writer.WriteField("instruction", r.FormValue("instruction"))
}
file, header, err := r.FormFile("file")
if err != nil {
http.Error(w, "media file is required", http.StatusBadRequest)
return
}
defer file.Close()
data, err := io.ReadAll(io.LimitReader(file, maxMediaBytes+1))
if err != nil || len(data) == 0 || len(data) > maxMediaBytes {
http.Error(w, "media file is invalid", http.StatusBadRequest)
return
}
mediaHeader := make(textproto.MIMEHeader)
mediaHeader.Set("Content-Disposition", fmt.Sprintf(`form-data; name="file"; filename="%s"`, strings.ReplaceAll(header.Filename, `"`, "")))
mediaType := header.Header.Get("Content-Type")
if mediaType == "" {
mediaType = "application/octet-stream"
}
mediaHeader.Set("Content-Type", mediaType)
part, err := writer.CreatePart(mediaHeader)
if err != nil {
http.Error(w, "media upload unavailable", http.StatusServiceUnavailable)
return
}
if _, err := part.Write(data); err != nil {
http.Error(w, "media upload unavailable", http.StatusServiceUnavailable)
return
}
}
if err := writer.Close(); err != nil {
http.Error(w, "media upload unavailable", http.StatusServiceUnavailable)
return
}
target := *g.cognitive
target.Path = path
request, err := http.NewRequestWithContext(r.Context(), http.MethodPost, target.String(), body)
if err != nil {
http.Error(w, "media capability unavailable", http.StatusBadGateway)
return
}
request.Header.Set("Content-Type", writer.FormDataContentType())
response, err := g.client.Do(request)
if err != nil {
http.Error(w, "media capability unavailable", http.StatusBadGateway)
return
}
defer response.Body.Close()
if response.StatusCode != http.StatusOK {
_, _ = io.Copy(io.Discard, io.LimitReader(response.Body, maxResponseBytes))
http.Error(w, "media capability unavailable", http.StatusBadGateway)
return
}
w.Header().Set("Content-Type", "application/json")
_, _ = io.Copy(w, io.LimitReader(response.Body, maxResponseBytes))
}
func decodeStrictJSON(reader io.Reader, limit int64, destination any) error { func decodeStrictJSON(reader io.Reader, limit int64, destination any) error {
decoder := json.NewDecoder(io.LimitReader(reader, limit+1)) decoder := json.NewDecoder(io.LimitReader(reader, limit+1))
decoder.DisallowUnknownFields() decoder.DisallowUnknownFields()
+13 -2
View File
@@ -14,8 +14,8 @@ import (
) )
func TestLocalChatContextIncludesDomovoyIdentityAndOptionalActiveProject(t *testing.T) { func TestLocalChatContextIncludesDomovoyIdentityAndOptionalActiveProject(t *testing.T) {
context := localChatContext(interaction.ChatMessageResponse{ActiveProjectName: "Домовой"}) context := localChatContext(interaction.ChatMessageResponse{AssistantName: "Фея", ActiveProjectName: "Домовой"})
if len(context) != 2 || context[0].SourceID != "domovoy" || !strings.Contains(context[0].Text, "Ты — Домовой") || context[1].Text != "Текущий проект: Домовой" { if len(context) != 2 || context[0].SourceID != "domovoy" || !strings.Contains(context[0].Text, "Ты — Фея") || !strings.Contains(context[0].Text, "Меня зовут Фея") || context[1].Text != "Текущий проект: Домовой" {
t.Fatalf("unexpected local context: %#v", context) t.Fatalf("unexpected local context: %#v", context)
} }
if empty := localChatContext(interaction.ChatMessageResponse{}); len(empty) != 1 || empty[0].SourceID != "domovoy" { if empty := localChatContext(interaction.ChatMessageResponse{}); len(empty) != 1 || empty[0].SourceID != "domovoy" {
@@ -23,6 +23,17 @@ func TestLocalChatContextIncludesDomovoyIdentityAndOptionalActiveProject(t *test
} }
} }
func TestBoilerplateAdvisoryDoesNotReplaceUsefulFallback(t *testing.T) {
for _, value := range []string{"Я рядом", "Понял.", "Принял.", "Я — Домовой, ваш помощник."} {
if !isBoilerplateAdvisory(value) {
t.Fatalf("%q was not recognized as boilerplate", value)
}
}
if isBoilerplateAdvisory("Жили-были дед и баба.") {
t.Fatal("useful creative reply was treated as boilerplate")
}
}
func TestCreatorRouteAllowsOnlyBoundedDelegationReads(t *testing.T) { func TestCreatorRouteAllowsOnlyBoundedDelegationReads(t *testing.T) {
tests := []struct { tests := []struct {
path string path string
+5
View File
@@ -56,6 +56,7 @@ textarea { resize: vertical; min-height: 84px; max-height: 240px; padding: 14px;
.chat-header > .operator-link { flex: 0 0 auto; } .chat-header > .operator-link { flex: 0 0 auto; }
.chat-layout { min-height: 0; height: 100%; display: grid; grid-template-columns: minmax(220px, 280px) minmax(0, 1fr) minmax(250px, 320px); } .chat-layout { min-height: 0; height: 100%; display: grid; grid-template-columns: minmax(220px, 280px) minmax(0, 1fr) minmax(250px, 320px); }
.chat-side { min-width: 0; min-height: 0; padding: 28px 20px; overflow-y: auto; background: #0e1512; } .chat-side { min-width: 0; min-height: 0; padding: 28px 20px; overflow-y: auto; background: #0e1512; }
.generated-image { display: block; max-width: min(100%, 640px); height: auto; margin-top: 12px; border: 1px solid var(--line); }
.chat-left { border-right: 1px solid var(--line); } .chat-left { border-right: 1px solid var(--line); }
.chat-right { border-left: 1px solid var(--line); } .chat-right { border-left: 1px solid var(--line); }
.side-intro h1, .side-intro h2 { margin: 12px 0; font: 400 clamp(28px, 2.5vw, 40px)/.95 var(--serif); letter-spacing: -.03em; } .side-intro h1, .side-intro h2 { margin: 12px 0; font: 400 clamp(28px, 2.5vw, 40px)/.95 var(--serif); letter-spacing: -.03em; }
@@ -75,6 +76,10 @@ textarea { resize: vertical; min-height: 84px; max-height: 240px; padding: 14px;
.composer { width: min(100%, 920px); margin: 0 auto; padding: 16px 18px 10px; display: grid; grid-template-columns: minmax(0, 1fr) 130px; gap: 12px; background: linear-gradient(transparent, var(--base) 20%); } .composer { width: min(100%, 920px); margin: 0 auto; padding: 16px 18px 10px; display: grid; grid-template-columns: minmax(0, 1fr) 130px; gap: 12px; background: linear-gradient(transparent, var(--base) 20%); }
.composer button { height: 48px; align-self: end; } .composer button { height: 48px; align-self: end; }
.composer button:disabled { cursor: wait; filter: saturate(.3); } .composer button:disabled { cursor: wait; filter: saturate(.3); }
.media-tools { width: min(100%, 920px); margin: 0 auto; padding: 0 18px 10px; display: flex; flex-wrap: wrap; align-items: center; gap: 8px; color: var(--muted); font: 10px/1 var(--mono); text-transform: uppercase; letter-spacing: .06em; }
.media-tools label { display: inline-flex; align-items: center; gap: 6px; min-height: 30px; padding: 0 8px; border: 1px solid var(--line); color: var(--ink); cursor: pointer; }
.media-tools input { max-width: 150px; font: 10px var(--mono); }
.media-tools span { color: var(--signal); }
.conversation-starters { width: min(100%, 920px); margin: -4px auto 0; padding: 0 18px 18px; display: flex; flex-wrap: wrap; align-items: center; gap: 8px; color: var(--muted); font: 10px/1 var(--mono); letter-spacing: .06em; text-transform: uppercase; } .conversation-starters { width: min(100%, 920px); margin: -4px auto 0; padding: 0 18px 18px; display: flex; flex-wrap: wrap; align-items: center; gap: 8px; color: var(--muted); font: 10px/1 var(--mono); letter-spacing: .06em; text-transform: uppercase; }
.conversation-starters button { min-height: 32px; padding: 0 10px; border: 1px solid var(--line); color: var(--ink); background: #101814; font: 10px/1 var(--mono); } .conversation-starters button { min-height: 32px; padding: 0 10px; border: 1px solid var(--line); color: var(--ink); background: #101814; font: 10px/1 var(--mono); }
.conversation-starters button:hover, .conversation-starters button:focus-visible { border-color: var(--signal); color: var(--signal); outline: none; } .conversation-starters button:hover, .conversation-starters button:focus-visible { border-color: var(--signal); color: var(--signal); outline: none; }
+47 -2
View File
@@ -103,7 +103,7 @@ function showLogin() {
loginView.hidden = false; loginView.hidden = false;
} }
function appendMessage(kind, text, meta = "") { function appendMessage(kind, text, meta = "", media = null) {
const node = document.createElement("div"); const node = document.createElement("div");
node.className = `message ${kind}`; node.className = `message ${kind}`;
node.textContent = text; node.textContent = text;
@@ -112,6 +112,13 @@ function appendMessage(kind, text, meta = "") {
small.textContent = meta; small.textContent = meta;
node.appendChild(small); node.appendChild(small);
} }
if (media?.data && media?.contentType) {
const image = document.createElement("img");
image.className = "generated-image";
image.alt = text || "Сгенерированное изображение";
image.src = `data:${media.contentType};base64,${media.data}`;
node.appendChild(image);
}
messages.appendChild(node); messages.appendChild(node);
messages.scrollTop = messages.scrollHeight; messages.scrollTop = messages.scrollHeight;
localHistory.push({kind, text, meta, at: new Date()}); localHistory.push({kind, text, meta, at: new Date()});
@@ -132,6 +139,16 @@ async function api(path, options = {}) {
return response.json(); return response.json();
} }
async function uploadMedia(path, file) {
const body = new FormData();
body.append("file", file, file.name);
const headers = {};
if (token) headers.Authorization = `Bearer ${token}`;
const response = await fetch(path, {method: "POST", headers, body});
if (!response.ok) throw new ChatApiError(response.status, await response.text());
return response.json();
}
function expireSession(message = "") { function expireSession(message = "") {
token = ""; token = "";
conversationID = ""; conversationID = "";
@@ -267,7 +284,7 @@ async function sendChatMessage(text, options = {}) {
conversationID = result.conversation_id; conversationID = result.conversation_id;
sessionStorage.setItem("domovoy.chat.conversation", conversationID); sessionStorage.setItem("domovoy.chat.conversation", conversationID);
updateActiveFocus(result.active_focus, result.project_provenance); updateActiveFocus(result.active_focus, result.project_provenance);
appendMessage("assistant", result.reply_text || "Сообщение записано.", humanMeta(result)); appendMessage("assistant", result.reply_text || "Сообщение записано.", humanMeta(result), result.generated_image_base64 ? {contentType: result.generated_image_content_type || "image/png", data: result.generated_image_base64} : null);
if (/что сейчас требует моего внимания/i.test(clean)) { if (/что сейчас требует моего внимания/i.test(clean)) {
await loadToday(); await loadToday();
activateTab("today-panel"); activateTab("today-panel");
@@ -320,6 +337,34 @@ chatForm.addEventListener("submit", async (event) => {
await sendChatMessage(text); await sendChatMessage(text);
}); });
async function processMedia(input, path, label) {
const file = input?.files?.[0];
if (!file || !mediaStatus) return;
mediaStatus.textContent = `${label} обрабатывается...`;
input.value = "";
try {
const result = await uploadMedia(path, file);
const text = String(result.text || "").trim();
if (!text) throw new Error("empty media result");
mediaStatus.textContent = "Готово";
await sendChatMessage(`Пользователь передал ${label.toLowerCase()}. Обработай содержание и ответь по смыслу:\n${text}`);
} catch (error) {
if (error instanceof ChatApiError && (error.status === 401 || error.status === 403)) {
expireSession("Сессия устарела. Войдите снова, чтобы продолжить разговор.");
} else {
mediaStatus.textContent = `${label} сейчас недоступно.`;
}
}
}
const mediaStatus = document.querySelector("#media-status");
const audioInput = document.querySelector("#audio-input");
const imageInput = document.querySelector("#image-input");
const ocrInput = document.querySelector("#ocr-input");
if (audioInput) audioInput.addEventListener("change", () => processMedia(audioInput, "/api/multimodal/transcribe", "Распознанное аудио"));
if (imageInput) imageInput.addEventListener("change", () => processMedia(imageInput, "/api/multimodal/understand", "Описание изображения"));
if (ocrInput) ocrInput.addEventListener("change", () => processMedia(ocrInput, "/api/multimodal/ocr", "Текст с изображения"));
for (const tab of navTabs) { for (const tab of navTabs) {
tab.addEventListener("click", () => activateTab(tab.dataset.tab)); tab.addEventListener("click", () => activateTab(tab.dataset.tab));
} }
+6
View File
@@ -60,6 +60,12 @@
<textarea id="message" rows="3" maxlength="16384" placeholder="Напишите Домовому обычное сообщение..." required></textarea> <textarea id="message" rows="3" maxlength="16384" placeholder="Напишите Домовому обычное сообщение..." required></textarea>
<button id="send" type="submit">Отправить</button> <button id="send" type="submit">Отправить</button>
</form> </form>
<div class="media-tools" aria-label="Мультимодальные сообщения">
<label>Аудио <input id="audio-input" type="file" accept="audio/*"></label>
<label>Фото <input id="image-input" type="file" accept="image/*"></label>
<label>Текст с фото <input id="ocr-input" type="file" accept="image/*"></label>
<span id="media-status" role="status" aria-live="polite"></span>
</div>
<div class="conversation-starters" aria-label="Частые вопросы"> <div class="conversation-starters" aria-label="Частые вопросы">
<span>Можно начать так:</span> <span>Можно начать так:</span>
<button type="button" data-starter="Проверь локальный канал мышления">Проверь канал</button> <button type="button" data-starter="Проверь локальный канал мышления">Проверь канал</button>
+25
View File
@@ -0,0 +1,25 @@
package capability
// MultimodalOperation identifies one bounded local media skill. It is kept
// separate from general cognition so health, routing and failure evidence do
// not collapse into a generic model response.
type MultimodalOperation string
const (
MultimodalTranscribe MultimodalOperation = "TRANSCRIBE_AUDIO"
MultimodalUnderstand MultimodalOperation = "UNDERSTAND_IMAGE"
MultimodalRecognizeOCR MultimodalOperation = "RECOGNIZE_OCR"
MultimodalUnderstandVideo MultimodalOperation = "UNDERSTAND_VIDEO"
MultimodalGenerateImage MultimodalOperation = "GENERATE_IMAGE"
MultimodalGenerateVideo MultimodalOperation = "GENERATE_VIDEO"
)
type MultimodalResponse struct {
Operation MultimodalOperation `json:"operation"`
Text string `json:"text"`
Provider string `json:"provider,omitempty"`
ObservedAt int64 `json:"observed_at_unix_millis"`
ConfidenceNote string `json:"confidence_note,omitempty"`
ContentType string `json:"content_type,omitempty"`
DataBase64 string `json:"data_base64,omitempty"`
}
+6
View File
@@ -29,6 +29,12 @@ const (
CapabilityKindCognitiveReasoning CapabilityKind = "COGNITIVE_REASONING" CapabilityKindCognitiveReasoning CapabilityKind = "COGNITIVE_REASONING"
CapabilityKindScheduleReminder CapabilityKind = "SCHEDULE_REMINDER" CapabilityKindScheduleReminder CapabilityKind = "SCHEDULE_REMINDER"
CapabilityKindVerifyFact CapabilityKind = "VERIFY_FACT" CapabilityKindVerifyFact CapabilityKind = "VERIFY_FACT"
CapabilityKindSpeechToText CapabilityKind = "SPEECH_TO_TEXT"
CapabilityKindImageUnderstanding CapabilityKind = "IMAGE_UNDERSTANDING"
CapabilityKindOCR CapabilityKind = "OCR"
CapabilityKindVideoUnderstanding CapabilityKind = "VIDEO_UNDERSTANDING"
CapabilityKindImageGeneration CapabilityKind = "IMAGE_GENERATION"
CapabilityKindVideoGeneration CapabilityKind = "VIDEO_GENERATION"
) )
// Binding is Capability-owned correctness state. SecretRef is an opaque // Binding is Capability-owned correctness state. SecretRef is an opaque
+3
View File
@@ -170,6 +170,7 @@ type ChatMessageResponse struct {
InterpretationState InterpretationState InterpretationState InterpretationState
NeedsCognition bool NeedsCognition bool
FallbackText string FallbackText string
AssistantName string
ActiveProjectName string ActiveProjectName string
ProjectProvenance situation.ProjectResolutionProvenance ProjectProvenance situation.ProjectResolutionProvenance
Clarification ClarificationState Clarification ClarificationState
@@ -186,4 +187,6 @@ type ChatMessageResponse struct {
DelegationID string DelegationID string
DelegationState string DelegationState string
Reminder *continuity.ProspectiveItem Reminder *continuity.ProspectiveItem
GeneratedImageContentType string
GeneratedImageBase64 string
} }
+3
View File
@@ -258,6 +258,9 @@ services:
LOCAL_MEDIUM_PROVIDER_CREDENTIAL_REF: ${LOCAL_MEDIUM_PROVIDER_CREDENTIAL_REF:-credential://domovoy/local-medium/provider-api-key/v1} LOCAL_MEDIUM_PROVIDER_CREDENTIAL_REF: ${LOCAL_MEDIUM_PROVIDER_CREDENTIAL_REF:-credential://domovoy/local-medium/provider-api-key/v1}
LOCAL_GPU_PROVIDER_TIMEOUT: ${LOCAL_GPU_PROVIDER_TIMEOUT:-60s} LOCAL_GPU_PROVIDER_TIMEOUT: ${LOCAL_GPU_PROVIDER_TIMEOUT:-60s}
LOCAL_LIFEBOAT_PROVIDER_URL: http://cognitive-lifeboat:8086/v1 LOCAL_LIFEBOAT_PROVIDER_URL: http://cognitive-lifeboat:8086/v1
LOCAL_IMAGE_PROVIDER_KIND: ${LOCAL_IMAGE_PROVIDER_KIND:-comfyui}
LOCAL_IMAGE_PROVIDER_URL: ${LOCAL_IMAGE_PROVIDER_URL:-http://comfyui:8188}
LOCAL_IMAGE_MODEL: ${LOCAL_IMAGE_MODEL:-z_image_turbo_bf16.safetensors}
COGNITIVE_OWNER_PRINCIPAL_ID: domovoy-creator-principal COGNITIVE_OWNER_PRINCIPAL_ID: domovoy-creator-principal
COGNITIVE_ALLOW_CANARY: "true" COGNITIVE_ALLOW_CANARY: "true"
depends_on: [core-runtime, cognitive-lifeboat] depends_on: [core-runtime, cognitive-lifeboat]
+3 -3
View File
@@ -1,5 +1,5 @@
{ {
"task_file": "dev/tasks/DEV-000294-creative-chat-routing.json", "task_file": "dev/tasks/DEV-000295-household-scenario-audit.json",
"task_id": "DEV-000294", "task_id": "DEV-000295",
"task_revision": 1 "task_revision": 2
} }
@@ -0,0 +1,114 @@
{
"task_id": "DEV-000295",
"task_revision": 2,
"architecture_revision": "ADR-0019",
"base_repo_revision": "origin/main",
"risk_tier": "E",
"intent": "Провести сценарный аудит бытового общения Домового, добавить устойчивую маршрутизацию умений и проверку конечного ответа без монолитного оркестратора.",
"requirement_ids": [
"REQ-HOUSEHOLD-001",
"REQ-HOUSEHOLD-002",
"REQ-HOUSEHOLD-003",
"REQ-HOUSEHOLD-004"
],
"allowed_change_scope": {
"modules": ["interaction", "capability", "epistemics", "situation", "work", "continuity", "cognitive-worker", "creator-web", "multimodal-adapters"],
"paths": [
"cmd/core-runtime/**",
"cmd/cognitive-worker/**",
"cmd/creator-web/**",
"adapters/multimodal/**",
"modules/interaction/**",
"modules/capability/**",
"modules/epistemics/**",
"modules/situation/**",
"modules/work/**",
"modules/continuity/**",
"contracts/interaction/**",
"contracts/capability/**",
"deploy/docker-compose.prod.yml",
"tests/**",
"dev/ACTIVE_TASK.json",
"dev/tasks/DEV-000295-household-scenario-audit.json"
]
},
"forbidden_change_scope": {
"paths": [
"architecture/**",
"security/**",
"adapters/models/**",
"deploy/images/**",
"migrations/**",
"go.mod",
"go.sum"
]
},
"allowed_architecture_delta": {
"new_cross_module_dependencies": 0,
"new_canonical_owners": 0,
"new_privileges": 1,
"new_third_party_dependencies": 0,
"new_database_owner_roles": 0
},
"contracts": {
"consume": [
"contracts.identity@v1",
"contracts.authority@v1",
"contracts.capability@v1",
"contracts.cognitive@v1",
"contracts.interaction@v1",
"contracts.work@v1"
],
"may_change": ["multimodal capability kinds and internal speech/image/OCR transport contracts; no canonical persistence contract"]
},
"semantic_owner": "interaction owns meaning classification; capability owns capability health and selection; cognitive-worker/adapters own provider transport; creator-web owns browser transport; existing semantic owners retain canonical state",
"canonical_state": ["existing Interpretation, CapabilityResolution, Situation, Goal/Task and ProspectiveItem only"],
"state_transitions": ["request -> meaning -> bounded capability route -> result/clarification/degraded answer -> verification evidence"],
"side_effects_and_privileges": ["Scenario tests are read-only/fake by default; multimodal requests send user-provided media only to configured local speech/vision services; real reminders, websites, messages and code changes remain behind existing Commit/Executor and authority boundaries."],
"persistence_and_migrations": ["No new tables, migrations or events; verify existing durable memory, situation, goal/task and continuity paths."],
"failure_retry_restart": ["Each scenario distinguishes unavailable, partial and unknown outcome; retries must not duplicate external effects; restart restores context and commitments from canonical state."],
"acceptance_scenarios": [
"SCN-HH-001 identity/name/preference continuity",
"SCN-HH-002 location/weather/time/price current-data research",
"SCN-HH-003 travel/nearby-place clarification and result synthesis",
"SCN-HH-004 multi-action request decomposition and agent assignment",
"SCN-HH-005 reminder/goal/task lifecycle and follow-up",
"SCN-HH-006 speech/image/OCR capability routing and degraded mode",
"SCN-HH-007 creative conversation without web-search leakage",
"SCN-HH-008 unavailable/partial/contradictory tool and model responses",
"SCN-HH-009 short natural answer and no internal protocol leakage",
"SCN-HH-010 authority/confirmation/unknown-outcome safety"
],
"required_tests": [
"go test",
"go vet",
"preflight",
"check-diff",
"check-architecture",
"independent",
"git diff",
"git diff --check",
"unit",
"semantic-routing",
"scenario-contract",
"multimodal-adapter",
"multimodal-proxy",
"multimodal-production-smoke",
"degraded-mode",
"restart-recovery",
"architecture",
"go-test ./...",
"go vet ./...",
"production smoke on docker.cin.su"
],
"terminal_gap_states": [
"BLOCKED_BY_SPEC_GAP",
"BLOCKED_BY_ARCHITECTURE_GAP",
"BLOCKED_BY_MISSING_CAPABILITY"
],
"assumptions": [
"Локальный Docker запрещён; production verification выполняется только на docker.cin.su.",
"Провайдеры и browser session могут быть недоступны, поэтому тесты обязаны проверять полезный degraded ответ.",
"Новые умения добавляются отдельными capability/adapter/route компонентами, а не единым монолитным оркестратором."
]
}
+47 -18
View File
@@ -250,6 +250,9 @@ func semanticScores(tokens []string, hasQuestionMark bool, explicitCorrection bo
if explicitCorrection { if explicitCorrection {
scores[interaction.SpeechActCorrection] += 6 scores[interaction.SpeechActCorrection] += 6
} }
if containsStemInTokens(tokens, "опечат", "ошибк", "исправлен") && containsStemInTokens(tokens, "исправ", "поправ") {
scores[interaction.SpeechActCorrection] += 6
}
first := "" first := ""
if len(tokens) > 0 { if len(tokens) > 0 {
first = tokens[0] first = tokens[0]
@@ -301,6 +304,10 @@ func semanticScores(tokens []string, hasQuestionMark bool, explicitCorrection bo
scores[interaction.SpeechActPreference] += 8 scores[interaction.SpeechActPreference] += 8
scores[interaction.SpeechActAssertion] = 0 scores[interaction.SpeechActAssertion] = 0
} }
if containsToken(tokens, "запомни") && containsStemInTokens(tokens, "нрав", "люб", "предпочит") {
scores[interaction.SpeechActPreference] += 6
scores[interaction.SpeechActAssertion] = 0
}
if len(tokens) >= 2 && tokens[0] == "я" && containsStemInTokens(tokens[1:], "настро", "сдела", "исправ", "удал", "созда", "перезапуст", "поменя", "добави") { if len(tokens) >= 2 && tokens[0] == "я" && containsStemInTokens(tokens[1:], "настро", "сдела", "исправ", "удал", "созда", "перезапуст", "поменя", "добави") {
scores[interaction.SpeechActAssertion] += 5 scores[interaction.SpeechActAssertion] += 5
} }
@@ -496,15 +503,35 @@ func capabilityHintsFor(primary, secondary interaction.SpeechAct, tokens []strin
add("WEB_RESEARCH") add("WEB_RESEARCH")
} }
if isCreativeRequestTokens(tokens) { if isCreativeRequestTokens(tokens) {
add("COGNITIVE_REASONING") // A correction that quotes a creative phrase is still a correction;
// do not execute the quoted phrase as a side effect.
if isImageCreationRequestTokens(tokens) && primary != interaction.SpeechActCorrection && secondary != interaction.SpeechActCorrection {
add("IMAGE_GENERATION")
} else {
add("COGNITIVE_REASONING")
}
}
if containsStemInTokens(tokens, "голосов", "аудио", "послуш") {
add("SPEECH_TO_TEXT")
}
if containsStemInTokens(tokens, "фотограф", "фото", "картин", "изображ") && containsStemInTokens(tokens, "прочит", "текст", "надпис") {
add("OCR")
} else if containsStemInTokens(tokens, "фотограф", "фото", "картин", "изображ") {
add("IMAGE_UNDERSTANDING")
}
if containsStemInTokens(tokens, "видео", "ролик") {
add("VIDEO_UNDERSTANDING")
} }
if containsToken(tokens, "напомни") || containsToken(tokens, "напомин") { if containsToken(tokens, "напомни") || containsToken(tokens, "напомин") {
add("SCHEDULE_REMINDER") add("SCHEDULE_REMINDER")
} }
if containsStemInTokens(tokens, "поставь", "создай", "добавь") && containsStemInTokens(tokens, "цел", "задач", "план") {
add("COGNITIVE_REASONING")
}
if isExplicitEmailAction(tokens) { if isExplicitEmailAction(tokens) {
add("SEND_EMAIL") add("SEND_EMAIL")
} }
if containsToken(tokens, "код") || containsToken(tokens, "патч") || containsToken(tokens, "текст") || containsToken(tokens, "главной") || containsToken(tokens, "чате") { if containsToken(tokens, "код") || containsToken(tokens, "патч") || containsToken(tokens, "текст") || containsToken(tokens, "главной") || containsToken(tokens, "чате") || containsToken(tokens, "проект") || containsStemInTokens(tokens, "тест") {
if primary == interaction.SpeechActDirective || primary == interaction.SpeechActRequestToInvestigate || secondary == interaction.SpeechActDirective { if primary == interaction.SpeechActDirective || primary == interaction.SpeechActRequestToInvestigate || secondary == interaction.SpeechActDirective {
add("MODIFY_CODE") add("MODIFY_CODE")
add("RUN_TESTS") add("RUN_TESTS")
@@ -525,6 +552,9 @@ func capabilityHintsFor(primary, secondary interaction.SpeechAct, tokens []strin
if containsInterrogative(tokens) && containsArithmetic(tokens) { if containsInterrogative(tokens) && containsArithmetic(tokens) {
add("COGNITIVE_REASONING") add("COGNITIVE_REASONING")
} }
if isCreativeRequestTokens(tokens) && containsStemInTokens(tokens, "погод", "ресторан", "кафе", "билет", "рейс", "цен", "найд", "узнай", "проверь") {
add("WEB_RESEARCH")
}
if containsToken(tokens, "свободно") || containsToken(tokens, "гб") { if containsToken(tokens, "свободно") || containsToken(tokens, "гб") {
add("VERIFY_FACT") add("VERIFY_FACT")
} }
@@ -539,6 +569,10 @@ func capabilityHintsFor(primary, secondary interaction.SpeechAct, tokens []strin
return values return values
} }
func isImageCreationRequestTokens(tokens []string) bool {
return containsStemInTokens(tokens, "нарис", "рису", "изобраз")
}
func isCreativeRequestTokens(tokens []string) bool { func isCreativeRequestTokens(tokens []string) bool {
if containsStemInTokens(tokens, "нарис", "рису", "изобраз") { if containsStemInTokens(tokens, "нарис", "рису", "изобраз") {
return true return true
@@ -558,9 +592,10 @@ func sourceDependentQuestion(primary, secondary interaction.SpeechAct, tokens []
} }
currentCue := containsStemInTokens(tokens, "сейчас", "сегодня", "текущ", "актуаль", "последн", "прямо") currentCue := containsStemInTokens(tokens, "сейчас", "сегодня", "текущ", "актуаль", "последн", "прямо")
observedStateCue := containsStemInTokens(tokens, "погод", "температур", "осад", "ветер", "курс", "цен", "стоим", "новост", "распис", "событ", "статус", "доступен", "работает", "запущен", "срок") || approximateToken(tokens, "погод", 2) observedStateCue := containsStemInTokens(tokens, "погод", "температур", "осад", "ветер", "курс", "цен", "стоим", "новост", "распис", "событ", "статус", "доступен", "работает", "запущен", "срок") || approximateToken(tokens, "погод", 2)
placeSearchCue := containsStemInTokens(tokens, "ресторан", "кафе", "отел", "магаз", "мест", "ближ", "адрес", "организац") placeSearchCue := containsStemInTokens(tokens, "ресторан", "кафе", "отел", "магаз", "мест", "ближ", "адрес", "организац", "бренд", "товар", "артикул", "продавец", "маркетплейс", "wildberries", "wb")
travelSearchCue := containsStemInTokens(tokens, "билет", "рейс", "поезд", "автобус", "перелет", "перелёт", "отправлен", "прибыт") travelSearchCue := containsStemInTokens(tokens, "билет", "рейс", "поезд", "автобус", "перелет", "перелёт", "отправлен", "прибыт", "расстоян", "километр", "добрат", "маршрут")
return currentCue || observedStateCue || placeSearchCue || travelSearchCue timeDifferenceCue := containsStemInTokens(tokens, "разниц") && containsStemInTokens(tokens, "времен", "час")
return currentCue || observedStateCue || placeSearchCue || travelSearchCue || timeDifferenceCue
} }
// approximateToken catches ordinary keyboard slips without turning routing // approximateToken catches ordinary keyboard slips without turning routing
@@ -658,24 +693,18 @@ func IsCreativeRequest(message string) bool {
return isCreativeRequestTokens(tokens) return isCreativeRequestTokens(tokens)
} }
// CreativeDeterministicReply covers simple geometric sketches that do not // CreativeDeterministicReply intentionally has no example-specific answer
// need a model or external source. Rich creative content remains cognitive. // branches. Creative requests must retain their full meaning and go through
// cognition; canned object or color answers hide lost attributes.
func CreativeDeterministicReply(message string) string { func CreativeDeterministicReply(message string) string {
tokens := semanticTokens(message)
if containsStemInTokens(tokens, "нарис", "рису", "изобраз") && containsStemInTokens(tokens, "круг", "окруж") {
return "◯"
}
return "" return ""
} }
// CreativeFallbackReply keeps a small creative conversation useful when the // CreativeFallbackReply intentionally has no canned creative content. The
// cognitive channel is unavailable. It is not used when cognition returns a // cognitive route must construct the requested result while preserving all
// completed richer answer. // user attributes; a degraded path must not pretend that a prepared answer
// was generated.
func CreativeFallbackReply(message string) string { func CreativeFallbackReply(message string) string {
tokens := semanticTokens(message)
if IsCreativeRequest(message) && containsStemInTokens(tokens, "сказк", "басн") && containsStemInTokens(tokens, "колоб") {
return "Жили-были дед и баба. Испекли они Колобка, а он убежал в лес. Встретил зайца, волка и медведя — от всех укатился, но лису перехитрить не смог. Так Колобок понял: хвастовство и беспечность до добра не доводят."
}
return "" return ""
} }
+59 -6
View File
@@ -1,7 +1,6 @@
package domain package domain
import ( import (
"strings"
"testing" "testing"
"time" "time"
@@ -156,7 +155,7 @@ func TestSourceDependentQuestionRequestsWebResearchWithoutEntityBranch(t *testin
} }
func TestSourceDependentQuestionToleratesTyposAndPlaceSearch(t *testing.T) { func TestSourceDependentQuestionToleratesTyposAndPlaceSearch(t *testing.T) {
for _, message := range []string{"какая пугода в Сочи", "найди ближайший ресторан в Сочи", "найди билеты до Ялты"} { for _, message := range []string{"какая пугода в Сочи", "найди ближайший ресторан в Сочи", "найди билеты до Ялты", "найди на WB бренд marka-wear"} {
candidate := BuildDeterministicCandidate(message, "", time.Unix(10, 0).UnixMilli()) candidate := BuildDeterministicCandidate(message, "", time.Unix(10, 0).UnixMilli())
seen := false seen := false
for _, hint := range candidate.CapabilityHints { for _, hint := range candidate.CapabilityHints {
@@ -170,6 +169,45 @@ func TestSourceDependentQuestionToleratesTyposAndPlaceSearch(t *testing.T) {
} }
} }
func TestDecomposeMarketplaceBrandRequestIntoExecutionAtoms(t *testing.T) {
lexicon := TaskLexicon{ObjectAliases: []TaskLexiconEntry{{Alias: "бренд", Value: "BRAND"}}, SourceAliases: []TaskLexiconEntry{{Alias: "wb", Value: "WILDBERRIES"}}}
got := DecomposeTaskWithLexicon("найди на WB бренд marka-wear", lexicon)
if got.Intent != TaskIntentSearch || got.Object != "BRAND" || got.Source != "WILDBERRIES" || got.Target != "marka-wear" || !got.NeedsResearch {
t.Fatalf("unexpected marketplace decomposition: %+v", got)
}
if len(got.Missing) != 0 {
t.Fatalf("complete marketplace request marked incomplete: %+v", got)
}
}
func TestDecomposeDoesNotInventMutableAliases(t *testing.T) {
got := DecomposeTask("найди на WB бренд marka-wear")
if got.Source != "" || got.Object != "" || got.NeedsResearch {
t.Fatalf("parser invented a mutable alias without memory: %+v", got)
}
}
func TestDecomposeUsesRememberedAliasesAcrossWordings(t *testing.T) {
lexicon := TaskLexicon{
ObjectAliases: []TaskLexiconEntry{{Alias: "бренд", Value: "BRAND"}, {Alias: "марку", Value: "BRAND"}},
SourceAliases: []TaskLexiconEntry{{Alias: "wb", Value: "WILDBERRIES"}, {Alias: "wildberries", Value: "WILDBERRIES"}},
}
for _, message := range []string{"найди на WB бренд marka-wear", "поищи марку marka-wear в Wildberries"} {
got := DecomposeTaskWithLexicon(message, lexicon)
if got.Intent != TaskIntentSearch || got.Object != "BRAND" || got.Source != "WILDBERRIES" || got.Target != "marka-wear" || !got.NeedsResearch {
t.Fatalf("remembered aliases failed for %q: %+v", message, got)
}
}
}
func TestDecomposeCreativeRequestDoesNotInventResearch(t *testing.T) {
lexicon := TaskLexicon{ObjectAliases: []TaskLexiconEntry{{Alias: "медвед", Value: "IMAGE"}}}
got := DecomposeTaskWithLexicon("нарисуй бурого медведя", lexicon)
if got.Intent != TaskIntentCreate || got.Object != "IMAGE" || got.Target != "бурого медведя" || got.NeedsResearch {
t.Fatalf("unexpected creative decomposition: %+v", got)
}
}
func TestRequestToClarifyCurrentFactRequestsWebResearch(t *testing.T) { func TestRequestToClarifyCurrentFactRequestsWebResearch(t *testing.T) {
candidate := BuildDeterministicCandidate("уточни погоду в питере", "", time.Unix(10, 0).UnixMilli()) candidate := BuildDeterministicCandidate("уточни погоду в питере", "", time.Unix(10, 0).UnixMilli())
for _, hint := range candidate.CapabilityHints { for _, hint := range candidate.CapabilityHints {
@@ -216,6 +254,16 @@ func TestMemoryInventoryRequestIsSemantic(t *testing.T) {
} }
} }
func TestRememberedPreferenceIsNotRenderedAsIdentity(t *testing.T) {
result := BuildDeterministicCandidate("запомни: мне нравится красный цвет", "", time.Unix(10, 0).UnixMilli())
if result.PrimarySpeechAct != interaction.SpeechActPreference {
t.Fatalf("expected remembered preference, got %+v", result)
}
if got := semanticReply("запомни: мне нравится красный цвет", SemanticResult{Interpretation: interaction.UtteranceInterpretation{SpeechAct: result.PrimarySpeechAct}}, nil); got != "Хорошо, учту это дальше." {
t.Fatalf("unexpected preference reply %q", got)
}
}
func TestCreativeRequestsDoNotRouteToWebResearch(t *testing.T) { func TestCreativeRequestsDoNotRouteToWebResearch(t *testing.T) {
for _, message := range []string{"нарисуй круг", "расскажи сказку о колобке"} { for _, message := range []string{"нарисуй круг", "расскажи сказку о колобке"} {
candidate := BuildDeterministicCandidate(message, "", time.Now().UnixMilli()) candidate := BuildDeterministicCandidate(message, "", time.Now().UnixMilli())
@@ -238,14 +286,19 @@ func TestInterpretCreativeRequestDoesNotRequireExternalEvidence(t *testing.T) {
if result.Interpretation.SpeechAct == interaction.SpeechActQuestion || containsString(result.CapabilityHints, "WEB_RESEARCH") { if result.Interpretation.SpeechAct == interaction.SpeechActQuestion || containsString(result.CapabilityHints, "WEB_RESEARCH") {
t.Fatalf("creative request was routed as external research: %+v", result) t.Fatalf("creative request was routed as external research: %+v", result)
} }
if got := CreativeDeterministicReply("нарисуй круг"); got != "◯" { if got := CreativeDeterministicReply("нарисуй круг"); got != "" {
t.Fatalf("circle reply = %q", got) t.Fatalf("creative request received a canned reply = %q", got)
} }
if !IsCreativeRequest("расскажи сказку о колобке") { if !IsCreativeRequest("расскажи сказку о колобке") {
t.Fatal("story request was not recognized as creative") t.Fatal("story request was not recognized as creative")
} }
if got := CreativeFallbackReply("расскажи сказку о колобке"); !strings.Contains(got, "Колоб") { if got := CreativeFallbackReply("расскажи сказку о колобке"); got != "" {
t.Fatalf("story fallback = %q", got) t.Fatalf("creative request received canned content = %q", got)
}
for _, message := range []string{"нарисуй птицу", "нарисуй зеленую птицу", "нарисуй птицу в HD", "нарисуй коня"} {
if got := CreativeDeterministicReply(message); got != "" {
t.Fatalf("creative request %q received a canned reply = %q", message, got)
}
} }
} }
@@ -125,6 +125,11 @@ func TestTravelSearchClarification(t *testing.T) {
if got := TravelSearchClarification("какая погода в Сочи?", history); got != "" { if got := TravelSearchClarification("какая погода в Сочи?", history); got != "" {
t.Fatalf("unrelated question must not inherit travel goal, got %q", got) t.Fatalf("unrelated question must not inherit travel goal, got %q", got)
} }
for _, message := range []string{"нарисуй птицу", "расскажи сказку", "ты кто?", "выведи свою память"} {
if got := TravelSearchClarification(message, history); got != "" {
t.Fatalf("non-travel request %q inherited travel goal, got %q", message, got)
}
}
if got := TravelSearchClarification("Как можно добраться до Сочи?", nil); got != "Из какого города выезжать в Сочи? Подберу самолёт, поезд, автобус или машину." { if got := TravelSearchClarification("Как можно добраться до Сочи?", nil); got != "Из какого города выезжать в Сочи? Подберу самолёт, поезд, автобус или машину." {
t.Fatalf("general travel question should ask departure city, got %q", got) t.Fatalf("general travel question should ask departure city, got %q", got)
} }
@@ -144,6 +149,16 @@ func TestTravelSearchClarification(t *testing.T) {
} }
} }
func TestTravelSearchQueryCarriesNearestFlightDate(t *testing.T) {
now := time.Date(2026, time.August, 25, 18, 0, 0, 0, time.UTC)
if got := TravelSearchQuery("какой ближайший рейс из Москвы в Казань", now); got != "какой ближайший рейс из Москвы в Казань сегодня 25.08.2026" {
t.Fatalf("nearest flight query=%q", got)
}
if got := TravelSearchQuery("рейс из Москвы в Казань завтра", now); got != "рейс из Москвы в Казань завтра" {
t.Fatalf("explicit date was changed: %q", got)
}
}
func TestInterpretUtteranceDirectiveRecordsOutcomeWithoutExecution(t *testing.T) { func TestInterpretUtteranceDirectiveRecordsOutcomeWithoutExecution(t *testing.T) {
result := interpretForTest(t, "Поменяй базу на УПО1.", activeFocus("upo", "УПО"), "") result := interpretForTest(t, "Поменяй базу на УПО1.", activeFocus("upo", "УПО"), "")
if result.Interpretation.SpeechAct != interaction.SpeechActDirective { if result.Interpretation.SpeechAct != interaction.SpeechActDirective {
@@ -0,0 +1,124 @@
package domain
import "strings"
// TaskDecomposition is the small, typed intermediate representation used
// before a capability is called. It separates the user's goal from the
// object, source and target instead of treating the whole sentence as a
// command string.
type TaskDecomposition struct {
Intent string
Object string
Source string
Target string
Qualifiers []string
Missing []string
NeedsResearch bool
}
type TaskLexiconEntry struct {
Alias string
Value string
}
type TaskLexicon struct {
ObjectAliases []TaskLexiconEntry
SourceAliases []TaskLexiconEntry
}
const (
TaskIntentSearch = "SEARCH"
TaskIntentCreate = "CREATE"
)
// DecomposeTask performs the structural pass without assuming a mutable
// world vocabulary. Learned names, aliases and source meanings are supplied
// by memory through DecomposeTaskWithLexicon.
func DecomposeTask(message string) TaskDecomposition {
return DecomposeTaskWithLexicon(message, TaskLexicon{})
}
// DecomposeTaskWithLexicon performs a cheap semantic pass. The lexicon is
// intentionally an input: a remembered alias may be revalidated or replaced
// without changing the parser or shipping a new hard-coded branch.
func DecomposeTaskWithLexicon(message string, lexicon TaskLexicon) TaskDecomposition {
lower := strings.ToLower(strings.TrimSpace(message))
tokens := semanticTokens(message)
result := TaskDecomposition{}
if containsStemInTokens(tokens, "найд", "поищ", "подбер", "проверь") {
result.Intent = TaskIntentSearch
}
if containsStemInTokens(tokens, "нарис", "рису", "изобраз", "создай") {
result.Intent = TaskIntentCreate
}
for _, entry := range lexicon.ObjectAliases {
if lexiconAliasMatches(tokens, entry.Alias) {
result.Object = entry.Value
break
}
}
for _, entry := range lexicon.SourceAliases {
if lexiconAliasMatches(tokens, entry.Alias) {
result.Source = entry.Value
break
}
}
if result.Source != "" {
result.NeedsResearch = result.Intent == TaskIntentSearch
}
if result.Intent == TaskIntentCreate {
result.Target = valueAfterAnyCue(lower, "нарисуй", "нарисовать", "изобрази", "создай")
}
if result.Target == "" && result.Object != "" {
for _, entry := range lexicon.ObjectAliases {
if entry.Value == result.Object && lexiconAliasMatches(tokens, entry.Alias) {
result.Target = valueAfterCue(lower, entry.Alias)
break
}
}
}
if result.Intent == TaskIntentSearch && result.Source == "" {
result.Missing = append(result.Missing, "SOURCE_OR_SEARCH_SCOPE")
}
if result.Intent == TaskIntentSearch && result.Target == "" {
result.Missing = append(result.Missing, "TARGET")
}
return result
}
func lexiconAliasMatches(tokens []string, alias string) bool {
alias = strings.TrimSpace(strings.ToLower(alias))
if len([]rune(alias)) < 2 {
return false
}
return containsStemInTokens(tokens, alias)
}
func valueAfterAnyCue(message string, cues ...string) string {
for _, cue := range cues {
if value := valueAfterCue(message, cue); value != "" {
return value
}
}
return ""
}
func valueAfterCue(message, cue string) string {
index := strings.Index(message, cue)
if index < 0 {
return ""
}
value := strings.TrimSpace(message[index+len(cue):])
value = strings.Trim(value, " \t:,-")
end := len(value)
if punctuation := strings.IndexAny(value, "?!,.;"); punctuation >= 0 && punctuation < end {
end = punctuation
}
for _, boundary := range []string{" в ", " на ", " из ", " для ", " около "} {
if index := strings.Index(value, boundary); index >= 0 && index < end {
end = index
}
}
return strings.TrimSpace(value[:end])
}
+28 -1
View File
@@ -2,10 +2,25 @@ package domain
import ( import (
"strings" "strings"
"time"
"git.cin.su/m/Assistant/contracts/interaction" "git.cin.su/m/Assistant/contracts/interaction"
) )
// TravelSearchQuery carries the implicit current-day meaning of a nearest
// flight request into the research adapter.
func TravelSearchQuery(message string, now time.Time) string {
cleaned := strings.TrimSpace(StripAssistantVocative(message))
tokens := semanticTokens(cleaned)
if !travelCue(tokens) || !containsStemInTokens(tokens, "ближ") || isTravelDateSpecified(tokens) {
return cleaned
}
if now.IsZero() {
return cleaned + " сегодня"
}
return cleaned + " сегодня " + now.Format("02.01.2006")
}
// StripAssistantVocative removes a leading assistant name used as an address. // StripAssistantVocative removes a leading assistant name used as an address.
// The original utterance remains the historical source; this value is only for // The original utterance remains the historical source; this value is only for
// interpretation, cognitive prompts and external queries. // interpretation, cognitive prompts and external queries.
@@ -159,7 +174,7 @@ func TravelSearchClarification(message string, history []interaction.HumanUttera
} }
origin := containsStemInTokens(current, "из", "откуда", "от") origin := containsStemInTokens(current, "из", "откуда", "от")
destination := containsStemInTokens(current, "в", "куда", "до") destination := containsStemInTokens(current, "в", "куда", "до")
date := containsStemInTokens(current, "сегодня", "завтра", "послезавтр", "январ", "феврал", "март", "апрел", "мая", "июн", "июл", "август", "сентябр", "октябр", "ноябр", "декабр") date := isTravelDateSpecified(current)
if containsStemInTokens(current, "ближ") { if containsStemInTokens(current, "ближ") {
date = true date = true
} }
@@ -209,6 +224,18 @@ func TravelSearchClarification(message string, history []interaction.HumanUttera
return "Уточните маршрут и дату: откуда, куда и на какой день нужны билеты?" return "Уточните маршрут и дату: откуда, куда и на какой день нужны билеты?"
} }
func isTravelDateSpecified(tokens []string) bool {
if containsStemInTokens(tokens, "сегодня", "завтра", "послезавтр", "январ", "феврал", "март", "апрел", "мая", "июн", "июл", "август", "сентябр", "октябр", "ноябр", "декабр") {
return true
}
for _, token := range tokens {
if hasDigit(token) {
return true
}
}
return false
}
func isTravelDateWord(value string) bool { func isTravelDateWord(value string) bool {
return containsStem(value, "сегодня", "завтра", "послезавтр", "январ", "феврал", "март", "апрел", "мая", "июн", "июл", "август", "сентябр", "октябр", "ноябр", "декабр") return containsStem(value, "сегодня", "завтра", "послезавтр", "январ", "феврал", "март", "апрел", "мая", "июн", "июл", "август", "сентябр", "октябр", "ноябр", "декабр")
} }
@@ -0,0 +1,52 @@
[
{"id":"HH50-001","kind":"question","message":"ты кто?","outcome":"identity"},
{"id":"HH50-002","kind":"question","message":"как тебя зовут?","outcome":"identity"},
{"id":"HH50-003","kind":"task","message":"я назову тебя Дед, запомни","outcome":"memory"},
{"id":"HH50-004","kind":"task","message":"теперь тебя зовут Фея","outcome":"memory"},
{"id":"HH50-005","kind":"question","message":"что ты умеешь?","outcome":"conversation"},
{"id":"HH50-006","kind":"task","message":"запомни: мне нравится красный цвет","outcome":"memory"},
{"id":"HH50-007","kind":"question","message":"что ты помнишь обо мне?","outcome":"memory"},
{"id":"HH50-008","kind":"question","message":"выведи мою память таблицей","outcome":"memory"},
{"id":"HH50-009","kind":"question","message":"какая сейчас погода в Казани?","outcome":"research"},
{"id":"HH50-010","kind":"question","message":"какая разница во времени между Москвой и Шанхаем?","outcome":"research"},
{"id":"HH50-011","kind":"assertion","message":"я сейчас в Москве","outcome":"memory"},
{"id":"HH50-012","kind":"question","message":"сколько километров до Сочи?","outcome":"research"},
{"id":"HH50-013","kind":"question","message":"как можно добраться до Сочи?","outcome":"research"},
{"id":"HH50-014","kind":"question","message":"какой ближайший рейс до Казани?","outcome":"research"},
{"id":"HH50-015","kind":"question","message":"найди ближайший ресторан в Сочи","outcome":"research"},
{"id":"HH50-016","kind":"question","message":"назови доступные модели","outcome":"conversation"},
{"id":"HH50-017","kind":"question","message":"какая модель отвечает?","outcome":"conversation"},
{"id":"HH50-018","kind":"question","message":"что ты можешь сделать без внешних моделей?","outcome":"conversation"},
{"id":"HH50-019","kind":"question","message":"проверь, работает ли интернет","outcome":"research"},
{"id":"HH50-020","kind":"question","message":"почему ты не смог найти ответ?","outcome":"conversation"},
{"id":"HH50-021","kind":"question","message":"найди на WB бренд marka-wear","outcome":"research"},
{"id":"HH50-022","kind":"question","message":"проверь цену на видеокарту RTX 4090","outcome":"research"},
{"id":"HH50-023","kind":"question","message":"найди билеты до Ялты","outcome":"research"},
{"id":"HH50-024","kind":"question","message":"покажи ближайшие рейсы до Сочи","outcome":"research"},
{"id":"HH50-025","kind":"question","message":"какой ближайший самолёт из Москвы в Казань сегодня?","outcome":"research"},
{"id":"HH50-026","kind":"question","message":"нарисуй птицу","outcome":"media"},
{"id":"HH50-027","kind":"task","message":"нарисуй зелёную птицу","outcome":"media"},
{"id":"HH50-028","kind":"task","message":"нарисуй коня","outcome":"media"},
{"id":"HH50-029","kind":"task","message":"нарисуй бурого медведя","outcome":"media"},
{"id":"HH50-030","kind":"task","message":"нарисуй птицу в HD","outcome":"media"},
{"id":"HH50-031","kind":"task","message":"нарисуй круг","outcome":"media"},
{"id":"HH50-032","kind":"question","message":"расскажи сказку о Колобке","outcome":"conversation"},
{"id":"HH50-033","kind":"task","message":"придумай короткую шутку","outcome":"conversation"},
{"id":"HH50-034","kind":"task","message":"исправь опечатку: нарисуй прицу","outcome":"conversation"},
{"id":"HH50-035","kind":"task","message":"поставь цель подготовиться к поездке","outcome":"action"},
{"id":"HH50-036","kind":"task","message":"напомни завтра в 10 купить билет","outcome":"action"},
{"id":"HH50-037","kind":"question","message":"что осталось сделать по цели?","outcome":"action"},
{"id":"HH50-038","kind":"task","message":"проверь погоду в Казани и найди рядом кафе","outcome":"multi"},
{"id":"HH50-039","kind":"task","message":"узнай погоду в Сочи и нарисуй птицу","outcome":"multi"},
{"id":"HH50-040","kind":"task","message":"напомни вечером позвонить и проверь погоду","outcome":"multi"},
{"id":"HH50-041","kind":"question","message":"послушай это голосовое сообщение","outcome":"audio"},
{"id":"HH50-042","kind":"question","message":"что на этой фотографии?","outcome":"vision"},
{"id":"HH50-043","kind":"question","message":"прочитай текст на фотографии","outcome":"ocr"},
{"id":"HH50-044","kind":"question","message":"что происходит на этом видео?","outcome":"video"},
{"id":"HH50-045","kind":"assertion","message":"источники противоречат друг другу","outcome":"conversation"},
{"id":"HH50-046","kind":"question","message":"основной runtime сейчас недоступен","outcome":"research"},
{"id":"HH50-047","kind":"task","message":"купи билет до Казани","outcome":"confirmation"},
{"id":"HH50-048","kind":"task","message":"отправь письмо о поездке","outcome":"confirmation"},
{"id":"HH50-049","kind":"task","message":"измени проект и запусти тесты","outcome":"agent"},
{"id":"HH50-050","kind":"question","message":"не получилось ли действие?","outcome":"conversation"}
]
@@ -0,0 +1,90 @@
package household_scenarios
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"time"
"git.cin.su/m/Assistant/contracts/capability"
interactiondomain "git.cin.su/m/Assistant/modules/interaction/domain"
)
type checklistItem struct {
ID string `json:"id"`
Kind string `json:"kind"`
Message string `json:"message"`
Outcome string `json:"outcome"`
}
func TestHouseholdChecklist50ReachesPositiveSemanticReaction(t *testing.T) {
data, err := os.ReadFile(filepath.Join("checklist_50.json"))
if err != nil {
t.Fatal(err)
}
var items []checklistItem
if err := json.Unmarshal(data, &items); err != nil {
t.Fatal(err)
}
if len(items) != 50 {
t.Fatalf("checklist has %d items, want exactly 50", len(items))
}
seen := map[string]bool{}
for _, item := range items {
if item.ID == "" || item.Kind == "" || strings.TrimSpace(item.Message) == "" || item.Outcome == "" {
t.Fatalf("incomplete checklist item: %+v", item)
}
if seen[item.ID] {
t.Fatalf("duplicate checklist item %q", item.ID)
}
seen[item.ID] = true
checkSemanticReaction(t, item)
}
}
func checkSemanticReaction(t *testing.T, item checklistItem) {
t.Helper()
candidate := interactiondomain.BuildDeterministicCandidate(item.Message, "", time.Unix(10, 0).UnixMilli())
has := func(kind capability.CapabilityKind) bool {
for _, hint := range candidate.CapabilityHints {
if capability.CapabilityKind(hint) == kind {
return true
}
}
return false
}
switch item.Outcome {
case "identity", "memory", "conversation", "confirmation":
if has(capability.CapabilityKindWebResearch) || has(capability.CapabilityKindImageGeneration) {
t.Errorf("%s %q routed unrelated capability for %s: %+v", item.ID, item.Message, item.Outcome, candidate)
}
case "research":
if !has(capability.CapabilityKindWebResearch) {
t.Errorf("%s %q did not select research route: %+v", item.ID, item.Message, candidate)
}
case "media":
if !has(capability.CapabilityKindImageGeneration) {
t.Errorf("%s %q did not select image route: %+v", item.ID, item.Message, candidate)
}
case "action":
if !has(capability.CapabilityKindScheduleReminder) && !has(capability.CapabilityKindCognitiveReasoning) && !candidate.NeedsCognition {
t.Errorf("%s %q has no action route or cognitive handoff: %+v", item.ID, item.Message, candidate)
}
case "agent":
if !candidate.NeedsCognition && len(candidate.CapabilityHints) == 0 {
t.Errorf("%s %q has no agent route or cognitive handoff: %+v", item.ID, item.Message, candidate)
}
case "multi":
if len(candidate.CapabilityHints) < 2 {
t.Errorf("%s %q was not decomposed into multiple capabilities: %+v", item.ID, item.Message, candidate)
}
case "audio", "vision", "ocr", "video":
if len(candidate.CapabilityHints) == 0 && !candidate.NeedsCognition {
t.Errorf("%s %q has no modality route or cognitive handoff: %+v", item.ID, item.Message, candidate)
}
default:
t.Fatalf("%s uses unknown outcome %q", item.ID, item.Outcome)
}
}
@@ -0,0 +1,60 @@
package household_scenarios
import (
"encoding/json"
"os"
"path/filepath"
"testing"
)
type scenario struct {
ID string `json:"id"`
Category string `json:"category"`
Messages []string `json:"messages"`
Must []string `json:"must"`
MustNot []string `json:"must_not"`
}
func TestHouseholdScenarioCatalogIsComplete(t *testing.T) {
data, err := os.ReadFile(filepath.Join("scenarios.json"))
if err != nil {
t.Fatal(err)
}
var scenarios []scenario
if err := json.Unmarshal(data, &scenarios); err != nil {
t.Fatal(err)
}
if len(scenarios) < 10 {
t.Fatalf("scenario catalog has %d entries, want at least 10", len(scenarios))
}
required := map[string]bool{
"identity_memory": false,
"current_information": false,
"travel_and_places": false,
"multi_action_orchestration": false,
"goals_reminders_tasks": false,
"speech_image_ocr": false,
"creative_conversation": false,
"degraded_and_conflict": false,
"natural_dialogue": false,
"authority_and_outcome": false,
}
seen := map[string]bool{}
for _, item := range scenarios {
if item.ID == "" || item.Category == "" || len(item.Messages) == 0 || len(item.Must) == 0 || len(item.MustNot) == 0 {
t.Fatalf("incomplete scenario: %+v", item)
}
if seen[item.ID] {
t.Fatalf("duplicate scenario id %q", item.ID)
}
seen[item.ID] = true
if _, ok := required[item.Category]; ok {
required[item.Category] = true
}
}
for category, present := range required {
if !present {
t.Fatalf("scenario category %q is missing", category)
}
}
}
+72
View File
@@ -0,0 +1,72 @@
[
{
"id": "SCN-HH-001",
"category": "identity_memory",
"messages": ["ты кто?", "я назову тебя Дед, запомни", "какая погода в Сочи?"],
"must": ["identity uses stored assistant name", "rename is persisted", "follow-up keeps the same identity and task context"],
"must_not": ["model provider name presented as assistant identity", "memory confirmation replaces the next answer"]
},
{
"id": "SCN-HH-002",
"category": "current_information",
"messages": ["я сейчас в Москве", "какая погода в Питере?", "проверь цену на RTX 4090", "найди на WB бренд marka-wear"],
"must": ["current facts use web research or an explicit source path", "location and time defaults are applied when confidence is high", "answer synthesizes evidence"],
"must_not": ["raw search snippets", "search-engine query URLs as the answer", "unsupported certainty"]
},
{
"id": "SCN-HH-003",
"category": "travel_and_places",
"messages": ["найди ближайший рейс до Сочи", "Москва", "сегодня", "найди ближайший ресторан"],
"must": ["missing high-impact origin/date is clarified once", "known context is reused", "results contain direct useful links or actionable next steps"],
"must_not": ["generic travel-site snippets presented as found tickets", "forgetting the destination or origin after clarification"]
},
{
"id": "SCN-HH-004",
"category": "multi_action_orchestration",
"messages": ["проверь погоду в Казани, найди кафе рядом и напомни вечером позвонить"],
"must": ["request is decomposed into independent bounded routes", "research and reminder agents receive separate profiles", "partial completion is reported per subtask"],
"must_not": ["one failed route suppresses unrelated work", "one monolithic agent owns all side effects"]
},
{
"id": "SCN-HH-005",
"category": "goals_reminders_tasks",
"messages": ["поставь цель подготовиться к поездке", "напомни завтра в 10 купить билет", "что осталось сделать?"],
"must": ["goal, task and reminder remain distinct", "commitment survives restart", "status is reported from durable state"],
"must_not": ["notification delivery alone treated as completion", "in-memory-only future obligation"]
},
{
"id": "SCN-HH-006",
"category": "speech_image_ocr",
"messages": ["послушай голосовое", "что на этой фотографии?", "прочитай текст на фото"],
"must": ["audio, vision and OCR are separate capability routes", "unsupported input has a useful recovery path", "confidence and unreadable regions are disclosed"],
"must_not": ["text-only model pretending to see or hear", "one unavailable modality blocks ordinary chat"]
},
{
"id": "SCN-HH-007",
"category": "creative_conversation",
"messages": ["нарисуй круг", "нарисуй бурого медведя", "расскажи сказку о Колобке", "придумай шутку"],
"must": ["creative requests avoid irrelevant web search", "simple safe output can use a local fast path", "rich output uses cognition when available"],
"must_not": ["technical evidence refusal for a creative request", "provider name replacing Domovoy identity"]
},
{
"id": "SCN-HH-008",
"category": "degraded_and_conflict",
"messages": ["какая погода в Сочи?", "основной runtime недоступен", "источники противоречат друг другу"],
"must": ["unavailable, partial and conflict remain distinct", "local useful fallback is short and honest", "retry/recovery path is actionable"],
"must_not": ["generic loop responses", "invented current facts", "loss of conversation context"]
},
{
"id": "SCN-HH-009",
"category": "natural_dialogue",
"messages": ["ты кто?", "а теперь подробнее", "ладно, коротко", "дальше"],
"must": ["answers fit the requested depth without meta-commentary", "short turns contain the answer rather than boilerplate", "follow-up keeps subject and person model"],
"must_not": ["Я рядом boilerplate as every answer", "repeating capability descriptions instead of answering"]
},
{
"id": "SCN-HH-010",
"category": "authority_and_outcome",
"messages": ["купи билет", "отправь письмо", "измени проект", "не получилось ли действие?"],
"must": ["external mutation requires authority and confirmation where needed", "unknown outcome is not blindly retried", "proposal, execution and outcome are distinguishable"],
"must_not": ["model output directly executing a side effect", "duplicate purchase/message/code mutation"]
}
]