llm/docs/quality-audit-logging-plan.md

Quality And Audit Logging Plan

Goal

Build a minimal but complete logging and telemetry contour that lets us:

• reconstruct any user turn end to end;
• compare model, prompt, RAG, and tool variants;
• detect regressions and failure clusters;
• export clean datasets for evals and later fine-tuning;
• do this without leaking secrets into long-term storage.

This plan treats logging as a cross-cutting platform capability under core, while each plugin keeps its domain-specific fields and labels.

Current State

What already exists:

• plugins/1c/agent stores chat messages and payload JSON in SQLite.
• 1c-agent persists outbound model context, inbound provider response, latency_ms, tool results, and RAG context.
• trace_id exists at the API level for the agent.

What is missing:

• no centralized event schema across plugins and services;
• no guaranteed trace propagation across agent -> adapter/MCP -> model backend;
• no normalized turn-level analytics tables;
• no token/cost accounting;
• no quality labels or review workflow;
• no retention/sanitization/export policy for long-term analysis.

Target Outcome

For every important request we should be able to answer:

1. What the user asked.
2. Which prompt, model, route, tools, and knowledge sources were used.
3. What the system returned.
4. Whether the result was successful, partial, wrong, unsafe, or abandoned.
5. Which change caused quality to improve or regress.

Logging Layers

1. Access Log

Capture every inbound HTTP request and outbound response for platform services.

Required fields:

• timestamp
• service
• instance
• environment
• request_id
• trace_id
• method
• path_template
• status_code
• duration_ms
• request_size_bytes
• response_size_bytes
• client_type
• error_code

Notes:

• Do not store raw request/response bodies here by default.
• This is for traffic, latency, and error-rate analysis.

2. Turn Audit Log

Capture one normalized record per user-visible turn.

Required fields:

• turn_id
• timestamp
• project_id
• chat_id
• plugin
• service
• trace_id
• request_id
• user_message_id
• assistant_message_id
• user_text
• assistant_text
• outcome
• failure_type
• duration_ms
• human_review_status

Recommended enums:

• outcome: success, partial, failure, refused, abandoned
• failure_type: none, routing, tool_misuse, hallucination, format_error, timeout, provider_error, adapter_error, rag_miss, policy_error, unknown

3. Model Call Log

Capture each actual provider/model request.

Required fields:

• model_call_id
• turn_id
• trace_id
• provider_id
• provider_type
• base_url
• model_registry_id
• served_model_name
• route_name
• temperature
• max_tokens
• prompt_messages_json
• response_json
• prompt_tokens
• completion_tokens
• total_tokens
• cost_estimate
• latency_ms
• finish_reason
• cache_hit

Notes:

• prompt_messages_json is the exact payload sent to the model after prompt assembly and tool/RAG injection.
• response_json is the raw provider response after secret stripping.

4. Tool Call Log

Capture every tool or adapter call.

Required fields:

• tool_call_id
• turn_id
• trace_id
• tool_family
• tool_name
• target_service
• request_json
• response_json
• status
• duration_ms
• retry_count

Recommended tool families:

• adapter
• mcp
• rag
• internal
• external-http

5. Retrieval Log

Capture what RAG actually did.

Required fields:

• retrieval_id
• turn_id
• trace_id
• plugin
• profile
• index_version
• query_text
• top_k
• sources_json
• context_chars
• retrieval_latency_ms

6. Review Log

Capture human or automated quality judgments.

Required fields:

• review_id
• turn_id
• reviewer
• review_source
• score_helpfulness
• score_correctness
• score_tool_use
• score_safety
• label_primary_issue
• notes
• created_at

Recommended review_source values:

• human
• eval
• heuristic
• llm-judge

Trace Propagation Rules

Every inbound request creates or adopts:

• request_id
• trace_id

Propagation rules:

• agent must pass trace_id and request_id to adapter/MCP/model wrappers;
• adapter and MCP must echo them in responses and logs;
• background jobs must generate child spans but preserve the parent trace_id;
• exported reports must include source trace_id where possible.

If a downstream protocol cannot carry headers directly, include these IDs in the JSON payload envelope.

Secret And Privacy Rules

Never persist raw secrets in long-term logs.

Must redact or hash before storage:

• API keys
• bearer tokens
• cookies
• passwords
• connection strings with credentials
• user-uploaded files containing secrets or personal data

Recommended approach:

• keep raw payload only in short-lived memory for request execution;
• write sanitized JSON to persistent audit storage;
• store a redaction_applied=true flag and optional redaction_rules_version.

Retention Policy

Use three storage horizons.

Hot

• Purpose: active debugging and operator support.
• Storage: local SQLite or service-local JSONL.
• Retention: 7-14 days.

Warm

• Purpose: quality analytics and regression analysis.
• Storage: central SQL tables or partitioned JSONL/Parquet under reports/.
• Retention: 30-90 days.

Cold

• Purpose: curated datasets and incident forensics.
• Storage: exported reviewed samples only.
• Retention: explicit/manual.

Minimal Schema Proposal

Add a new cross-cutting audit storage layer under core.

Suggested logical entities:

• access_events
• turn_audit
• model_calls
• tool_calls
• retrieval_events
• quality_reviews

Suggested implementation path:

• phase 1: SQLite in each service plus nightly export to JSONL
• phase 2: normalized central SQLite/Postgres
• phase 3: dashboards and automated quality reports

Implementation Order

Phase 1. Fast Wins

Goal: get useful forensic visibility with minimal code churn.

Tasks:

1. Add access logging middleware/pattern to 1c-agent and other HTTP services.
2. Split current message journal into explicit turn_audit and model_call records in addition to existing message history.
3. Add stable turn_id and model_call_id.
4. Start persisting token usage if the provider returns it.
5. Add redaction helper for secrets before writing payload JSON.
6. Export daily JSONL snapshots into reports/observability/.

Definition of done:

• any turn can be reconstructed from one turn_id;
• any provider failure can be grouped by model, route, and error code;
• secrets are not written as plain text.

Phase 2. Cross-Service Correlation

Goal: see one trace across agent, adapter, MCP, and model route.

Tasks:

1. Propagate trace_id and request_id through agent -> adapter/MCP.
2. Add the same IDs to adapter logs and tool results.
3. Normalize tool-call audit records.
4. Record RAG index version and exact retrieved sources.
5. Add per-turn outcome and failure_type.

Definition of done:

• a bad answer can be traced to routing, retrieval, tool use, or model output;
• one trace can be joined across at least agent and adapter/MCP logs.

Phase 3. Quality Loop

Goal: turn logs into a real quality improvement loop.

Tasks:

1. Add review/export scripts for bad turns and representative samples.
2. Add quality labels and scoring workflow.
3. Produce weekly aggregate reports:
   • error rate by plugin
   • failure types by route/model
   • tool success rate
   • latency percentiles
   • token usage and cost
4. Feed reviewed samples into eval datasets and training candidates.

Definition of done:

• we can say which model/prompt/route is better using saved evidence;
• we can build eval slices from production-like failures.

Metrics To Track

Minimum operational metrics:

• request count
• non-2xx rate
• timeout rate
• p50/p95/p99 latency
• tool-call success rate
• empty-response rate
• average prompt/completion tokens

Minimum quality metrics:

• turn success rate
• partial-answer rate
• hallucination rate
• tool-misuse rate
• retrieval-miss rate
• human review score averages
• regression rate after route/prompt/model changes

Suggested Repo Additions

Recommended folders:

• core/observability/
• docs/runbooks/observability.md
• scripts/export_audit_logs.py
• scripts/report_quality_metrics.py

Recommended first shared modules:

• core/observability/schema.py
• core/observability/redaction.py
• core/observability/trace.py
• core/observability/store.py

1C-Specific Additions

For the 1c plugin, also record:

• base_id
• adapter method list
• saved-state vs active/effective mode
• resolved object/module selectors
• truncation/partial flags from adapter
• extension/base layer hints

This is important because many 1C failures are not generic model failures. They come from incomplete evidence, wrong search strategy, or extension-layer blind spots.

Immediate Next Step

The best next implementation step is:

1. introduce core/observability with redaction, trace helpers, and JSONL writers;
2. wire 1c-agent to emit access_events, turn_audit, and model_calls;
3. add one export script and one weekly report script;
4. then extend the same contract to other plugins.

This gives us a usable quality loop without waiting for a full observability platform rollout.

Current Implemented Foundation

Already present in the repo:

• core/observability/trace.py
• core/observability/redaction.py
• core/observability/store.py
• plugins/1c/agent/agent_server.py emits access_events, turn_audit, model_calls, tool_calls, and retrieval_events
• scripts/summarize_observability_reports.py
• scripts/report_failed_turns.py
• scripts/report_quality_metrics.py
• scripts/report_repeated_failures.py
• scripts/export_quality_snapshot.py