wip(dev-000073): preserve durable bootstrap checkpoint
CI / test (push) Failing after 19s

DEV-000073 is NOT DONE; current work is partial/stabilized WIP only. go test ./cmd/devctl was last verified PASS. Remaining bootstrap/secret integration requires a stronger long-horizon model. This commit is preservation-only; current ancestry includes DEV-000072 candidate 36fec6c05cdf7aaf413fe... and must be reconciled before DEV-000073 promotion.
This commit is contained in:
m
2026-08-18 12:07:11 +03:00
parent 36fec6c05c
commit 754cbe43d2
12 changed files with 2679 additions and 5 deletions
@@ -0,0 +1,169 @@
# DEV-000073 Deployment Continuity Evidence (Checkpoint)
Audit date: 2026-08-18
Task ID: `DEV-000073`
Task revision: `1`
Base revision: `36fec6c05cdf7aaf413ffe6b9437f07b03fc7b6c`
## Status summary
- `DEV_000073_STATUS = BLOCKED`
- `CURRENT_PHASE = DURABLE_BOOTSTRAP_PARTIAL_IMPLEMENTATION_STABILIZED`
- `CHECKPOINT_STATUS = IN_PROGRESS`
- `LAST_VERIFIED_COMMIT = 36fec6c05cdf7aaf413ffe6b9437f07b03fc7b6b`
- `LAST_VERIFIED_TEST_COMMAND = go test ./cmd/devctl`
- `LAST_VERIFIED_TEST_RESULT = EXECUTED_PASS`
- `CURRENT_UNCOMMITTED_DIFF = cmd/devctl/deploy.go, cmd/devctl/deploy_test.go, docs/DEPLOYMENT.md, AGENTS.md, architecture/UNRESOLVED_QUESTIONS.md, dev/tasks/DEV-000073-production-deployment-continuity-safe-redeploy.json, docs/evidence/deployment-continuity-dev000073.md, docs/evidence/production-recovery-incident-20260818.md, docs/PROJECT_STATUS.md, dev/ACTIVE_TASK.json`
- `CURRENT_TASK_STATE = PAUSED_PENDING_LONG_HORIZON_MODEL`
## Executed checks
- `go test ./cmd/devctl` → `EXECUTED_PASS`
- `go run ./cmd/devctl preflight -task dev/tasks/DEV-000073-production-deployment-continuity-safe-redeploy.json` → `EXECUTED_PASS`
- `go run ./cmd/devctl check-diff -task dev/tasks/DEV-000073-production-deployment-continuity-safe-redeploy.json` → `EXECUTED_PASS`
- `go test ./cmd/devctl` targeted acceptance set (`TestDeployPrepareBlockedAndApplyRefusesMissingRequiredInput`, `TestAssertPlanCurrentRejectsStaleInputChange`, `TestAssertPlanCurrentRejectsArtifactMismatch`, `TestDeploymentPlanDoesNotLeakSecretValues`, `TestDeployApplyRunsHealthGateWithReadyPlanAndSimulatedComposeDocker`, `TestDeployApplyHealthGateRejectsOnFailedHTTPHealth`, migration precondition tests) → `EXECUTED_PASS`
## Task-specific acceptance probes
- `MISSING_INPUT_NEGATIVE_TEST = EXECUTED_PASS`
- missing required input in prepare → `readiness=BLOCKED`
- `deploy apply` with blocked plan returns non-zero
- `STALE_PLAN_NEGATIVE_TEST = EXECUTED_PASS`
- compose/input/content assumption changes in plan → `assertPlanCurrent` detects stale state
- `EXACT_ARTIFACT_NEGATIVE_TEST = EXECUTED_PASS`
- artifact change without plan refresh → `assertPlanCurrent` detects stale mismatch
- `SECRET_LEAK_TEST = EXECUTED_PASS`
- plan output and tests show no plaintext secret values; only source class and hashes persisted
- `HEALTH_GATE_TEST = EXECUTED_PASS`
- docker compose success + passing HTTP checks + docker inspect => apply may complete
- docker compose success + failed `/healthz` => apply fails
## Production continuity probe (sanitized shell)
- `PRODUCTION_REVISION_VERIFIED = 11a6a23f6e4f44dd88265a04846419a27f54df98`
- `REAL_PRODUCTION_PREPARE_READINESS = BLOCKED`
- `REAL_PRODUCTION_SANITIZED_PREPARE = NOT_READY`
- `REAL_PRODUCTION_CONFIG_SOURCE = TEMPORARY_SYNTHETIC`
- `DOCKER_PROD_BOOTSTRAP_ENV_SOURCE = secrets/docker-prod-bootstrap.env`
- `LEGACY_AMBIENT_DEPENDENCIES = none`
- `UNKNOWN_INPUTS = none`
- `KNOWN_INPUT_SOURCE_DETAIL` for required fields:
- `ASSISTANT_SOURCE_REVISION = missing`
- `CREATOR_WEB_PORT = missing`
- `NOTIFICATION_RECEIVER_PORT = missing`
- `POSTGRES_DB = file`
- all resolved/required production inputs include:
- resolved from file: `POSTGRES_SUPERUSER`, `POSTGRES_SUPERUSER_PASSWORD`, `CONTINUITY_DB_PASSWORD`, `WORK_DB_PASSWORD`, `CAPABILITY_DB_PASSWORD`, `COMMIT_DB_PASSWORD`, `AUTHORITY_DB_PASSWORD`, `IDENTITY_DB_PASSWORD`, `SITUATION_DB_PASSWORD`, `CREATOR_USERNAME`, `COMMIT_EXECUTOR_TOKEN`
- missing (blocking): `CREATOR_BOOTSTRAP_TOKEN`, `CREATOR_PASSWORD_FILE`, `CREATOR_SESSION_SIGNING_KEY`, `EPISTEMICS_POSTGRES_CREDENTIAL_REF`, `EPISTEMICS_POSTGRES_PASSWORD`, `INTERACTION_POSTGRES_CREDENTIAL_REF`, `INTERACTION_POSTGRES_PASSWORD`
## Configuration source classification (required production inputs)
- `POSTGRES_SUPERUSER -> BOOTSTRAP` (from env file path)
- `POSTGRES_SUPERUSER_PASSWORD -> SECRET_REFERENCE` (from env file path)
- `CONTINUITY_DB_PASSWORD -> SECRET_REFERENCE` (from env file path)
- `WORK_DB_PASSWORD -> SECRET_REFERENCE` (from env file path)
- `CAPABILITY_DB_PASSWORD -> SECRET_REFERENCE` (from env file path)
- `COMMIT_DB_PASSWORD -> SECRET_REFERENCE` (from env file path)
- `AUTHORITY_DB_PASSWORD -> SECRET_REFERENCE` (from env file path)
- `IDENTITY_DB_PASSWORD -> SECRET_REFERENCE` (from env file path)
- `SITUATION_DB_PASSWORD -> SECRET_REFERENCE` (from env file path)
- `CREATOR_USERNAME -> BOOTSTRAP` (from env file path)
- `CREATOR_PASSWORD_FILE -> DOMOVOY_MANAGED` (missing)
- `CREATOR_BOOTSTRAP_TOKEN -> SECRET_REFERENCE` (missing)
- `CREATOR_SESSION_SIGNING_KEY -> SECRET_REFERENCE` (missing)
- `COMMIT_EXECUTOR_TOKEN -> SECRET_REFERENCE` (from env file path)
- `EPISTEMICS_POSTGRES_CREDENTIAL_REF -> SECRET_REFERENCE` (missing)
- `INTERACTION_POSTGRES_CREDENTIAL_REF -> SECRET_REFERENCE` (missing)
- `EPISTEMICS_POSTGRES_PASSWORD -> SECRET_REFERENCE` (missing)
- `INTERACTION_POSTGRES_PASSWORD -> SECRET_REFERENCE` (missing)
- `CONFIG_SOURCE_CLASSIFICATION = TEMPORARY_SYNTHETIC`
## Migration readiness
- `CURRENT_MIGRATION_STATE_SOURCE = git ls-tree -r --name-only <revision> database/migrations`
- `CANDIDATE_MIGRATION_REQUIREMENT = current_max=27 target_max=27 status=matched need_known_good=false (for plan created from 11a6a23f6e4f...)`
- `MIGRATION_DISCOVERY_STATUS = PASS`
- `PRECONDITION_ENFORCEMENT = PASS`
- `MIGRATION_PRECONDITION_STATUS = PASS` (plan execution enforces migration precondition and rejects upgrade-without-known-good as blocking)
- `ROLLBACK_COMPATIBILITY_STATUS = not reached by this evidence run; migration check is enforced before compose mutation`
## Gaps / blockers
- `BLOCKER = REAL_PRODUCTION_SANITIZED_PREPARE depends on `secrets/docker-prod-bootstrap.env` and still has 7 required missing inputs`
- `BLOCKER = production bootstrap source is temporary/synthetic in this environment; no durable/protected production bootstrap source has been established here`
- `REASON_FOR_PAUSE = remaining work requires long-horizon model for durable bootstrap review, resolver bridge completion, and production seed migration planning`
## Evidence persisted
- `evidence/evidence file: docs/evidence/deployment-continuity-dev000073.md`
- `checkpoint: docs/PROJECT_STATUS.md`
## Durable Ownership Audit (DEV-000073)
INPUT_OWNERSHIP_MATRIX
| INPUT_NAME | WHY_NEEDED | SEMANTIC_OWNER | CANONICAL_STATE | SECRET | INTENDED_DURABLE_SOURCE | CURRENT_ACTUAL_SOURCE | RESOLUTION_PATH | WHY_CURRENT_PREPARE_CANNOT_RESOLVE_IT |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CREATOR_BOOTSTRAP_TOKEN | bootstrap token for the Creator domain startup/authentication bootstrap flow | identity | yes | yes | durable secret input for production bootstrap (protected secret value) | missing from sanitized workspace prepare inputs | deploy.prepare file/env resolution only (`inputSourceClass=SECRET_REFERENCE`) | no durable bootstrap record supplied for this token in this environment; prepare does not have a secret backend lookup step |
| CREATOR_PASSWORD_FILE | creator password secret-file locator consumed by core runtime identity startup | identity | no | no | domovoy-managed runtime mount under `/run/secrets` (`core-runtime` creator password path) | provided in compose as a fixed container path but not as resolved prepare input | deploy.prepare accepts `_FILE` as an input class but only from env file/ambient and does not consume compose defaults | compose default is outside deploy.prepare source model, so required input remains unresolved even when container expects the path |
| CREATOR_SESSION_SIGNING_KEY | key for secure creator session signing | identity | yes | yes | durable secret input for production bootstrap (protected secret value) | missing from sanitized workspace prepare inputs | deploy.prepare file/env resolution only (`inputSourceClass=SECRET_REFERENCE`) | no durable bootstrap source in this environment; no value in env file and not provided by resolver bootstrap path |
| EPISTEMICS_POSTGRES_CREDENTIAL_REF | identity of epistemics credential record for DB secret resolution | epistemics | yes | no | protected durable secret-reference declaration for the epistemics DB credential record | missing from sanitized workspace prepare inputs | deploy.prepare expects direct env value (`SECRET_REFERENCE`) | credential reference value is absent and there is no durable bootstrap source for this reference name |
| EPISTEMICS_POSTGRES_PASSWORD | DB password transport input currently listed as required by deploy | epistemics | no | yes | derived at runtime from `EPISTEMICS_POSTGRES_CREDENTIAL_REF` + `EPISTEMICS_POSTGRES_PASSWORD_FILE` through local secret resolver | not present as input in prepare; compose sets file-based runtime path for password value | unresolved by deploy.prepare because it is treated as legacy required input, while runtime reads file-backed path | this value is rendered runtime transport data; deploy.prepare contract has no mapping from `*_CREDENTIAL_REF` to `*_PASSWORD` |
| INTERACTION_POSTGRES_CREDENTIAL_REF | identity of interaction credential record for DB secret resolution | interaction | yes | no | protected durable secret-reference declaration for the interaction DB credential record | missing from sanitized workspace prepare inputs | deploy.prepare expects direct env value (`SECRET_REFERENCE`) | credential reference value is absent and there is no durable bootstrap source for this reference name |
| INTERACTION_POSTGRES_PASSWORD | DB password transport input currently listed as required by deploy | interaction | no | yes | derived at runtime from `INTERACTION_POSTGRES_CREDENTIAL_REF` + `INTERACTION_POSTGRES_PASSWORD_FILE` through local secret resolver | not present as input in prepare; compose sets file-based runtime path for password value | unresolved by deploy.prepare because it is treated as legacy required input, while runtime reads file-backed path | this value is rendered runtime transport data; deploy.prepare contract has no mapping from `*_CREDENTIAL_REF` to `*_PASSWORD` |
CREDENTIAL_REF_PASSWORD_DUPLICATION_ASSESSMENT
`B) CredentialRef is canonical and PASSWORD is derived/rendered runtime transport value.`
Rationale:
- `cmd/core-runtime/main.go` `readCredentialRefSecret(name, refName)` rejects direct `name` secret values (`direct environment secret is forbidden`) and requires `name_FILE` + `<name>_CREDENTIAL_REF`.
- It resolves credentials through `platformsecrets.NewFileResolver` using `SecretFileRef{CredentialRef: ref, Purpose: DatabasePassword}`.
- The compose layout includes `<OWNER>_POSTGRES_CREDENTIAL_REF` and `<OWNER>_POSTGRES_PASSWORD_FILE`, reinforcing the canonical-reference + rendered-file pattern.
CREATOR_INPUT_OWNERSHIP
- `CREATOR_BOOTSTRAP_TOKEN`
- Real owner: `identity`.
- Real durable source class: protected secret (identity bootstrap token), **not** in repo and not present in sanitized prepare input.
- Current classification: missing required input in this environment.
- `CREATOR_SESSION_SIGNING_KEY`
- Real owner: `identity`.
- Real durable source class: protected secret (identity session signing key), **not** in repo and not present in sanitized prepare input.
- Current classification: missing required input in this environment.
- `CREATOR_PASSWORD_FILE`
- Real owner: `identity` (runtime auth material locator).
- It is a **rendered runtime path** backed by Domovoy/runtime secret mount conventions (`/run/secrets`) and is **not** a canonical secret value.
- Current classification: compose-fixed path, not a standalone durable deploy input for this phase.
DOCKER_PROD_BOOTSTRAP_ENV_ROLE
`TEMPORARY_SYNTHETIC`
Why: the file is a local repository bootstrap artifact used for sanitized shell-level prepare/apply probing; it is not an intentional durable/protected production source and still lacks required durable secret inputs. It is neither a Domovoy-managed state artifact nor a recoverable provenance artifact for real production bootstrap.
EXISTING_SECRET_RESOLVER_CAN_SUPPLY_MISSING_INPUTS
`partial`
- adapter? `no` (deploy.prepare does not invoke runtime secret adapter; this check is done in runtime after plan creation)
- bootstrap locator? `missing` (no durable, environment-wide bootstrap secret locator for all 7 inputs is present in this workspace)
- credential record? `partially present in contracts, not provisioned here` (code path expects `*_PASSWORD_FILE` + `*_CREDENTIAL_REF`, but missing refs/records in this environment)
- protected backend entry? `missing` (no proven production-grade protected entry available via the current prepare evidence run)
- ownership mapping? `partial` (owners/areas can be inferred from contracts and module code, but prepare-time ownership policy is not the missing-input resolver)
- pre-runtime bootstrap cycle? `not demonstrated` (no cycle blocker)
BOOTSTRAP_CYCLE_EXISTS
`no`
No real cycle was found because secret resolution uses local file-backed resolver in core runtime and is not a dependency of Domovoy boot for deployment plan creation in this repository evidence path.
MINIMUM_IMPLEMENTATION_GAP
- Provide a durable/protected bootstrap source for the remaining required identity inputs (`CREATOR_BOOTSTRAP_TOKEN`, `CREATOR_SESSION_SIGNING_KEY`, `CREATOR_PASSWORD_FILE`) and epistemics/interaction credential refs.
- Align deploy input contract so production input readiness depends on the canonical pair inputs (`*_CREDENTIAL_REF` + `*_PASSWORD_FILE`) and not legacy `<OWNER>_POSTGRES_PASSWORD` as a required direct input.
- Add evidence-backed mapping for each canonical input’s ownership/source class and bootstrap provisioning path before proceeding to acceptance/ready status.
CHECKPOINT_UPDATED = yes
EVIDENCE_PERSISTED = yes
@@ -0,0 +1,82 @@
## Production Recovery Incident Log
Date: 2026-08-18
Objective: Restore last known-good production with minimal changes, no code deploy from DEV-000072.
### 1) Pre-recovery verification (reconfirmed)
- Last observed bad running revision: `36fec6c4f8aabf5e9f5f4ac0d5e2f9c0b8a5a9f0`.
- Last known-good revision from durable manifests: `11a6a23f6e4f44dd88265a04846419a27f54df98`.
- Known-good deployment artifact IDs were present in `docker-prod` and matched both:
- `%LOCALAPPDATA%\Domovoy\monitoring\deployment-manifest.json`
- `\\nas\MST\codex\_monitoring\domovoy\deployment-manifest.json`
- Pre-recovery container state:
- `assistant-postgres-1` was running but unhealthy.
- `assistant-notification-receiver-1` healthy.
- `assistant-core-runtime-1`, `assistant-commit-runtime-1`, `assistant-cognitive-worker-1`, `assistant-creator-web-1` were `Created`.
- `assistant-continuity-runtime-1` was restarting.
- Continuity startup logs repeatedly reported `COMMIT_EXECUTOR_TOKEN is required`.
- DB read checks were reachable through `postgres` role and owner roles existed; schema objects in main owner schemas were present, including `interaction.utterance_interpretations` + `interaction.semantic_candidates` and expected 4 semantic columns.
### 2) Recovered configuration inputs
Classification used:
- `DURABLE_CANONICAL`: long-lived non-secret production evidence.
- `PROTECTED_DURABLE`: host/docker secret artifacts that are persistent, permission-protected inputs.
- `BREAK_GLASS_RUNTIME_EVIDENCE`: local runtime incident recovery metadata used only after verification.
- `POSTGRES_SUPERUSER` / passwords:
- `DURABLE_CANONICAL`: compose contract requires them.
- `PROTECTED_DURABLE`: secret-backed postgres volumes for password material.
- `BREAK_GLASS_RUNTIME_EVIDENCE`: `secrets/production-recovery.env` provided non-empty values and enabled rollback configuration.
- `COMMIT_EXECUTOR_TOKEN`:
- `DURABLE_CANONICAL`: required by continuity/commit contracts.
- `BREAK_GLASS_RUNTIME_EVIDENCE`: populated from `secrets/production-recovery.env`.
- `CREATOR_USERNAME`:
- `BREAK_GLASS_RUNTIME_EVIDENCE`: `secrets/production-recovery.env` (`m`).
- `epistemics_db_password`, `interaction_db_password`, `creator_bootstrap_token`, `creator_password`, `creator_session_signing_key`,
`notification writer/reader tokens`:
- `PROTECTED_DURABLE`: existing Docker volumes `assistant_core_secrets`,
`assistant_notification_writer_secret`, `assistant_notification_reader_secret`.
### 3) Recovery plan (finalized)
- Target revision: `11a6a23f6e4f44dd88265a04846419a27f54df98`.
- Keep durable data volumes unchanged:
- `assistant_assistant_pg`
- `assistant_core_secrets`
- `assistant_notification_writer_secret`
- `assistant_notification_reader_secret`
- `assistant_notification_receipts`
- `assistant_nats`
- Action set:
1. Retag known-good images from local/NAS manifests as active `assistant-* : latest`.
2. Recreate assistant stack on `docker-prod` with `--env-file secrets/production-recovery.env`, `--no-build`, `--force-recreate`.
3. Verify health and endpoint checks listed in section 5.
### 4) Recovery actions executed
- Recreated all assistant services on `docker-prod` (and `nats`) from known-good image IDs.
- Loaded production recovery env (`secrets/production-recovery.env`) into compose execution.
- Confirmed required runtime env is now present in service env:
- continuity/commit `COMMIT_EXECUTOR_TOKEN` is set.
- db user/passwords + creator bootstrap wiring are set.
- Verified all services now run with revision `11a6a23f6e4f44dd88265a04846419a27f54df98`.
- No credential values were printed in logs or evidence text.
### 5) Verification results
- `POSTGRES_HEALTH`: OK (postgres accepts connections).
- `CORE_RUNTIME_HEALTH`: OK (`/healthz` returns runtime ok).
- `CONTINUITY_RUNTIME_HEALTH`: OK (`/healthz` returns runtime ok).
- `COMMIT_RUNTIME_HEALTH`: OK (`/healthz` returns runtime ok).
- `CREATOR_WEB_HEALTH`: OK (`/healthz` returns runtime ok; container healthy).
- `EXTERNAL_HEALTH`: `https://assistant.cin.su/healthz` returns `HTTP 200`.
- `CHAT_SMOKE`: `https://assistant.cin.su/chat` returns `HTTP 200`; authenticated smoke message sent successfully via `POST /api/chat/messages` after session creation (harmless message `health check`), returning `HTTP 200`.
- Production artifact confirmation succeeded for revision `11a6a23f6e4f44dd88265a04846419a27f54df98` with 7 services.
### 6) Final state
- `KNOWN_GOOD_IMAGE_STATUS`: all required known-good image IDs present and active.
- `PRODUCTION_REVISION_AFTER_RECOVERY`: `11a6a23f6e4f44dd88265a04846419a27f54df98`.
- `DATA_INTEGRITY_STATUS`: preserved; existing schemas and `interaction.utterance_interpretations` semantic columns remain present.
- `CREDENTIAL_ROTATIONS`: none (used recovered durable config; no new credentials generated).
- Remaining limitation: commit/execution of this recovery changed image tags and container recreate timing only; no code/database migrations were altered.
### 7) Task status after recovery
- `DEV-000072_STATUS`: `PRODUCTION_ACCEPTANCE_BLOCKED_BY_DEPLOYMENT_INCIDENT` remains.
- Continue production readiness requires normal restart of production acceptance run from `docs/PROJECT_STATUS` after this incident checkpoint.