2761 lines
184 KiB
Markdown
2761 lines
184 KiB
Markdown
# CODEX CONTEXT
|
|
|
|
## Project identity
|
|
|
|
This project is a production-grade distributed secure access platform.
|
|
|
|
It started as a custom RDP proxy with persistent server-side sessions, but the final target architecture is broader:
|
|
|
|
- distributed secure access fabric
|
|
- multi-tenant platform
|
|
- session broker for GUI and future non-GUI protocols
|
|
- cluster mesh of nodes
|
|
- connector/VPN layer
|
|
- customer-managed and platform-managed nodes
|
|
- node-agent based self-update / rollback / health supervision
|
|
|
|
## Product architecture rule: VPN and Remote Workspace are separate products/layers
|
|
|
|
Do not merge VPN/IP tunnel work with Remote Workspace / remote desktop work.
|
|
|
|
- VPN is a universal network-layer IP tunnel. It carries any traffic generated
|
|
by a phone, Windows PC, Linux host, or other client device: HTTP, DNS, ping,
|
|
RDP clients, SSH clients, SMB, business apps, and future protocols. VPN must
|
|
stay protocol-agnostic and must not contain remote-desktop-specific logic.
|
|
- Remote Workspace is an application/session-layer service. The client talks to
|
|
RAP using RAP's own client protocol. RAP workers/connectors then talk to the
|
|
target server using protocol adapters such as RDP, SSH, VNC, or future
|
|
adapters, convert screen/input/clipboard/files/audio/control into RAP's
|
|
format, and render it in the RAP client.
|
|
- VPN optimization work must focus on generic data-plane transport,
|
|
full-tunnel/split-tunnel routing, DNS, MTU/MSS, QoS, NAT traversal, direct
|
|
UDP/QUIC transport, fallback relay, diagnostics, and stability for arbitrary
|
|
traffic.
|
|
- Remote Workspace optimization work must focus on server catalog, session
|
|
broker, workers/connectors, protocol adapters, RAP client protocol, separate
|
|
connection windows, rendering/input/clipboard/file/audio behavior, and
|
|
user-facing remote-workspace UX.
|
|
- Both VPN and Remote Workspace must consume the shared Fabric Service Channel
|
|
runtime. Control/API traffic may use backend/admin ingress, but working
|
|
service data must use the fabric channel whenever available. Backend relay is
|
|
a compatibility/degraded fallback, not the production steady-state.
|
|
- The accepted service-channel direction is documented in
|
|
`docs/architecture/FABRIC_SERVICE_CHANNEL_RUNTIME.md`: a service requests a
|
|
channel with entry pool, exit pool, roles, service class, channel classes,
|
|
QoS and failover policy; the fabric selects the fastest healthy route and
|
|
rebuilds it on failure. Protocol-specific services must not reimplement this
|
|
transport.
|
|
- Current implementation: backend issues `rap.fabric_service_channel_lease.v1`
|
|
leases and embeds them in VPN client profiles. Leases include
|
|
cluster-authority-signed `rap.fabric_service_channel_lease_authority.v1`
|
|
payloads that bind token hash, selected route, generation, fencing epoch, and
|
|
expiry, plus a signed `data_plane` contract declaring that working data uses
|
|
the Fabric Service Channel over fabric routes while backend relay is only an
|
|
explicit degraded/disabled fallback policy. `rap-node-agent` accepts the
|
|
first VPN packet service-channel entry
|
|
endpoint under
|
|
`/api/v1/clusters/{cluster_id}/fabric/service-channels/{channel_id}/vpn-connections/{resource_id}/packets`
|
|
plus `/packets/ws`. The endpoint validates the signed or introspected
|
|
data-plane contract, applies the preferred fabric route, uses the existing
|
|
production `vpn_packet` fabric route, reports contract adoption in heartbeat
|
|
access telemetry, and refuses backend relay when the contract disables it.
|
|
Backend access telemetry and web-admin now show data-plane adoption,
|
|
working/steady-state transport, backend relay policy, data-plane mode, and
|
|
logical flow mode at cluster/node/channel levels. The next slice is explicit
|
|
route/fallback violation incidents from that telemetry, plus client
|
|
consumption of the lease endpoint template.
|
|
|
|
## Current proven foundation
|
|
|
|
The current codebase already proved the most risky low-level lifecycle assumptions for RDP:
|
|
|
|
- real FreeRDP connect works
|
|
- session state transitions to active work
|
|
- terminate works
|
|
- detach works without killing the remote session
|
|
- reattach works without recreating the remote session
|
|
- takeover works without recreating the remote session
|
|
- per-resource certificate verification policy exists
|
|
- `certificate_verification_mode = strict | ignore`
|
|
- `strict` is default
|
|
- `ignore` works on a per-resource basis
|
|
- worker build is reproducible
|
|
- backend build is reproducible
|
|
|
|
This proven lifecycle must NOT be broken by future architecture work.
|
|
|
|
## Current architecture baseline
|
|
|
|
Current audit and baseline snapshot:
|
|
|
|
- `docs/audits/PROJECT_AUDIT_2026-04-26.md`
|
|
- `docs/audits/CURRENT_BASELINE_MATRIX.md`
|
|
|
|
### Test environment
|
|
- Canonical test Docker host: `192.168.200.61`
|
|
- Canonical Docker context: `test-ubuntu`
|
|
- Canonical SSH alias: `docker-test`
|
|
- Current external control-plane endpoint for remote/offsite node enrollment:
|
|
`http://94.141.118.222:19191` / `http://vpn.cin.su:19191`.
|
|
- Current port forward: `94.141.118.222:19191` -> `192.168.200.61:18080`.
|
|
- For offsite Windows/Linux nodes, install profiles should use:
|
|
`http://vpn.cin.su:19191/api/v1` as control-plane endpoint and
|
|
`http://vpn.cin.su:19191/downloads` as artifact endpoint unless the user
|
|
explicitly chooses the raw IP endpoint.
|
|
- Backend API for local/client smoke runs: `http://192.168.200.61:8080/api/v1`
|
|
- WebSocket gateway for local/client smoke runs: `ws://192.168.200.61:8080/api/v1/gateway/ws`
|
|
- Stage C17 planning is completed.
|
|
- C17A synthetic mesh runtime skeleton is implemented and test-proven in
|
|
`rap-node-agent` only. It is disabled by default and carries synthetic
|
|
`fabric.probe` / `fabric.probe_ack` messages only.
|
|
- C17B route health and failover probes are implemented and test-proven in
|
|
`rap-node-agent` only. They are disabled by default and carry synthetic
|
|
`fabric.route_health` / `fabric.route_health_ack` messages only.
|
|
- C17C relay semantic hardening is implemented and test-proven in
|
|
`rap-node-agent` only. It is disabled by default and models synthetic
|
|
per-channel queues/QoS/backpressure only.
|
|
- C17D non-production test-service path is implemented and test-proven in
|
|
`rap-node-agent` only. It is disabled by default and carries only bounded
|
|
`synthetic.echo` test payloads.
|
|
- C17E/C17F/C17G are implemented and proven for live synthetic HTTP transport,
|
|
scoped synthetic route config, and Control Plane scoped synthetic config
|
|
consumption.
|
|
- C17H deployed multi-agent synthetic config smoke is runtime-proven on
|
|
`docker-test`: five running `rap-node-agent` containers consume
|
|
backend-issued node-scoped synthetic config, direct and single-relay
|
|
synthetic route-health observations return to the Control Plane, and
|
|
production forwarding remains disabled.
|
|
- C17I production forwarding gate foundation is implemented and test-proven:
|
|
`rap-node-agent` has an explicit production-forwarding gate, while
|
|
`/mesh/v1/forward` still refuses production payload forwarding until a later
|
|
approved runtime stage.
|
|
- C17J production envelope contract is implemented and test-proven:
|
|
`/mesh/v1/forward` validates route-bound production envelopes for
|
|
`fabric_control` / `fabric.control` only when the gate is enabled, rejects
|
|
service channels, and still refuses production forwarding.
|
|
- C17K production envelope observation is implemented and test-proven:
|
|
valid accepted envelopes can be observed locally as metadata-only records
|
|
after validation; rejected envelopes are not observed, observation failure
|
|
fails closed, and production forwarding remains unavailable.
|
|
- C17L bounded production observation sink is implemented and test-proven:
|
|
accepted metadata-only observations can be retained locally with fixed
|
|
capacity, oldest-entry drop behavior, and no payload body storage.
|
|
- C17M production observation sink wiring is implemented and test-proven:
|
|
node-agent can wire the bounded local metadata-only sink when
|
|
`RAP_MESH_PRODUCTION_OBSERVATION_SINK_CAPACITY` is explicitly greater than
|
|
zero; the wiring is disabled by default and exposes no read API.
|
|
- C17N production observation sink metrics are implemented and test-proven:
|
|
local sink metrics expose only capacity, current depth, accepted total, and
|
|
dropped-oldest total; they expose no observation records or payload metadata.
|
|
- C17O production observation sink local metrics logging is implemented and
|
|
test-proven: node-agent logs aggregate sink metrics locally when the sink is
|
|
explicitly enabled; no read API or Control Plane reporting is added.
|
|
- C17P production observation sink change-driven metrics logging is implemented
|
|
and test-proven: node-agent suppresses repeated identical local sink metrics
|
|
logs; no read API or Control Plane reporting is added.
|
|
- C17Q production forwarding gate/runtime log boundary is implemented and
|
|
test-proven: node-agent logs production forwarding gate state separately from
|
|
production forwarding runtime state. Runtime state remained false until
|
|
C17Z introduced gate-controlled `fabric.control` direct forwarding.
|
|
- C17R production observation sink capacity guard is implemented and
|
|
test-proven: `RAP_MESH_PRODUCTION_OBSERVATION_SINK_CAPACITY` is rejected
|
|
above `10000`.
|
|
- C17S production observation panic fail-closed hardening is implemented and
|
|
test-proven: observer errors and observer panics both fail closed as
|
|
observation failure.
|
|
- C17T production envelope payload boundary is implemented and test-proven:
|
|
validated production `fabric.control` envelope payloads are bounded to
|
|
`4096` bytes and oversized envelopes are rejected before observation.
|
|
- C17U production envelope created-at skew boundary is implemented and
|
|
test-proven: validated production `fabric.control` envelopes whose
|
|
`created_at` is more than one minute in the future are rejected before
|
|
observation.
|
|
- C17V peer endpoint candidate model is implemented and test-proven:
|
|
node-scoped synthetic mesh config now carries route-scoped endpoint
|
|
candidates with transport, address, reachability, NAT type, connectivity
|
|
mode, priority, policy tags, verification time, and metadata. This is a
|
|
model/config boundary only; no production route scoring, NAT traversal,
|
|
shortcut routing, or forwarding runtime is implemented.
|
|
- C17W peer endpoint candidate scoring model is implemented and test-proven:
|
|
`rap-node-agent` can rank already-scoped endpoint candidates using soft
|
|
inputs such as transport, reachability, connectivity mode, NAT type,
|
|
priority, region, policy tags, channel class, and verification age. This is
|
|
a scoring helper only; it does not open connections, choose production
|
|
routes, or forward payloads.
|
|
- C17X health-aware endpoint candidate scoring overlay is implemented and
|
|
test-proven: endpoint candidate scoring can optionally use local health
|
|
observations keyed by `endpoint_id`, including latency, success/failure
|
|
history, recent failure reason, reliability score, and observation freshness.
|
|
This remains advisory scoring only and is not wired into production route
|
|
execution.
|
|
- C17Y Platform Owner synthetic mesh visibility is implemented and
|
|
build/test-proven: `web-admin` reads node-scoped synthetic mesh config and
|
|
shows config enabled state, route counts, peer endpoints, endpoint
|
|
candidates, C17X advisory scoring boundary, and `production_forwarding`.
|
|
This remains platform-owner visibility only and does not enable production
|
|
forwarding.
|
|
- C17Z production fabric-control direct forwarding boundary is implemented and
|
|
test-proven: when `RAP_MESH_PRODUCTION_FORWARDING_ENABLED=true`,
|
|
`/mesh/v1/forward` can deliver valid route-bound `fabric.control` envelopes
|
|
at the local destination or forward them to a direct next hop from explicit
|
|
peer endpoint config. Service channels, arbitrary relay forwarding,
|
|
multi-hop production route execution, and RDP/VPN/file/video/service payloads
|
|
remain unavailable.
|
|
- C17Z1 production fabric-control multi-hop route-path boundary is implemented
|
|
and test-proven: production `fabric.control` envelopes can carry
|
|
`route_path` and `visited_node_ids`; relay nodes validate path position,
|
|
forward only to the next path node, update TTL/hop/visited metadata, and
|
|
reject loops. Service payloads remain unavailable.
|
|
- C17Z2 production fabric-control forwarding observability boundary is
|
|
implemented and test-proven: node-agent emits local
|
|
`mesh_production_forward_event` logs for accepted, forwarded, delivered, and
|
|
rejected production `fabric.control` envelopes. Logs are metadata-only and
|
|
include no payload bodies or read API.
|
|
- C17Z3 production fabric-control route-config boundary is implemented and
|
|
test-proven: when scoped/control-plane mesh routes are available locally,
|
|
production `fabric.control` envelopes must match configured route_id/path/
|
|
next-hop/channel/expiry/TTL/hop limits before forwarding.
|
|
- C17Z4 scoped peer directory and recovery seeds boundary is implemented and
|
|
test/build-proven: node-scoped mesh config carries scoped `peer_directory`
|
|
and explicit bounded `recovery_seeds`; node-agent parses/validates them and
|
|
web-admin shows counts.
|
|
- C17Z5 node-agent peer cache runtime boundary is implemented and test-proven:
|
|
node-agent builds a local `PeerCache`, selects bounded warm peers, probes warm
|
|
peers with `/mesh/v1/health`, and reports metadata-only mesh-link
|
|
observations when synthetic mesh testing is enabled.
|
|
- C17Z6 dynamic endpoint reporting boundary is implemented and test-proven:
|
|
node-agent reports explicit advertised mesh endpoint metadata in heartbeat,
|
|
and Control Plane projects latest reported endpoints/candidates into
|
|
node-scoped synthetic mesh config.
|
|
- C17Z7 private/corporate endpoint candidate boundary is implemented and
|
|
test-proven: node-agent reports multiple advertised endpoint candidates,
|
|
scoring rewards private/corporate same-site candidates, and peer cache can
|
|
use the best candidate address for warm health.
|
|
- C17Z8 peer connection state machine boundary is implemented and test-proven:
|
|
node-agent tracks warm-peer states `disconnected`, `connecting`, `ready`,
|
|
`degraded`, and `backoff`, with bounded backoff after repeated health probe
|
|
failures.
|
|
- C17Z9 peer recovery planner boundary is implemented and test-proven:
|
|
node-agent targets a bounded stable ready-peer set, enters recovery when
|
|
ready peers fall below target, and selects bounded recovery probes from warm
|
|
peers, recovery seeds, and other connectable scoped peers.
|
|
- C17Z10 peer connection intent planner boundary is implemented and
|
|
test-proven: node-agent classifies bounded peer work as maintain/probe/
|
|
recover and classifies transport readiness as direct/private_lan/
|
|
corporate_lan/outbound_only/relay_required, with rendezvous-required
|
|
metadata only.
|
|
- C17Z11 peer connection manager runtime boundary is implemented and
|
|
test-proven: node-agent uses a reusable HTTP keep-alive client for real
|
|
control-plane health probes of direct/private/corporate peers and records
|
|
`waiting_rendezvous` for outbound-only/relay-required peers.
|
|
- C17Z12 rendezvous/relay control-plane contract is implemented and
|
|
docker-test-runtime-proven: backend issues node-scoped `rendezvous_leases`,
|
|
node-agent resolves matching `waiting_rendezvous` intents into
|
|
`relay_control`, probes relay `/mesh/v1/health`, records and maintains
|
|
`relay_ready`, and keeps service payload forwarding disabled.
|
|
- C17Z13 rendezvous lease telemetry is implemented and
|
|
docker-test-runtime-proven: node-agent reports
|
|
`mesh_rendezvous_lease_report` with relay admission, peer admission,
|
|
TTL/renewal posture, `relay_ready`, and explicit no-payload boundary flags;
|
|
web-admin shows `rv leases` in recent heartbeat tables.
|
|
- C17Z14 rendezvous lease refresh contract is implemented and
|
|
docker-test-runtime-proven: node-agent refreshes renewal-needed/stale
|
|
rendezvous leases through node-scoped synthetic config reload, updates the
|
|
running peer cache/route/lease state, and reports refresh plus stale relay
|
|
withdrawal/reselection telemetry. Service payload forwarding remains
|
|
unavailable.
|
|
- C17Z15 backend relay replacement policy is implemented and
|
|
docker-test-runtime-proven: backend consumes recent stale-relay heartbeat
|
|
feedback, withdraws stale explicit rendezvous leases, scores alternate relay
|
|
candidates from route adjacency, endpoint priority, policy tags, and recent
|
|
mesh-link health, and returns replacement leases plus
|
|
`rendezvous_relay_policy` decisions in node-scoped synthetic config.
|
|
Node-agent reports `c17z15.mesh_rendezvous_lease_report.v1` and keeps stale
|
|
state scoped to the exact lease/relay, so replacement leases for the same
|
|
peer are not marked stale by association. Service payload forwarding remains
|
|
unavailable.
|
|
- C17Z16 route/path decision artifact is implemented and
|
|
docker-test-runtime-proven: backend `c17z16.synthetic.v1` config includes
|
|
`route_path_decisions` with original hops, effective hops, local previous/
|
|
next hop, selected replacement relay, generation, score reasons, and
|
|
no-payload boundary flags. Node-agent stores the control-plane route
|
|
generation and reports `c17z16.mesh_route_path_decision_report.v1` plus
|
|
`c17z16.mesh_rendezvous_lease_report.v1`. Service payload forwarding remains
|
|
unavailable.
|
|
- C17Z17 node-side route generation tracker is implemented and
|
|
docker-test-runtime-proven: backend `c17z17.synthetic.v1` config and
|
|
node-agent `mesh_route_generation_report` track active/applied/unchanged/
|
|
withdrawn route decisions, generation changes, total counters, and
|
|
`withdrawn_by_replacement` records for stale relay paths when replacement is
|
|
first observed. Service payload forwarding remains unavailable.
|
|
- C17Z18 synthetic route-health effective path runtime is implemented and
|
|
docker-test-runtime-proven: backend `c17z18.synthetic.v1` config and
|
|
node-agent `mesh_route_health_config_report` apply Control Plane
|
|
`route_path_decisions` to synthetic route-health route config only. The
|
|
synthetic runtime probes selected effective paths through replacement relays,
|
|
reports expected/observed hops and drift state, and backend latest mesh links
|
|
preserve route-health observations separately from connection-manager
|
|
observations. Service payload forwarding remains unavailable.
|
|
- C17Z19 synthetic route-health feedback scoring is implemented and
|
|
docker-test-runtime-proven: backend consumes recent `synthetic_route_health`
|
|
observations in relay scoring, uses drift/unreachable/failure metadata to
|
|
mark the exact selected relay stale, boosts healthy low-latency relay
|
|
candidates, and returns replacement leases/route decisions through the
|
|
existing synthetic config contract. Migration `000022` adds the `synthetic`
|
|
mesh service class. Service payload forwarding remains unavailable.
|
|
- C17Z20 node-side route-health feedback refresh is implemented and
|
|
docker-test-runtime-proven: after reporting synthetic route-health
|
|
drift/unreachable/failure, node-agent performs a bounded node-scoped
|
|
synthetic-config refresh, applies returned replacement route decisions to
|
|
route-health config immediately, and reports
|
|
`c17z20.mesh_route_health_feedback_refresh_report.v1`. Service payload
|
|
forwarding remains unavailable.
|
|
- C17Z21 offsite control-plane bootstrap relay and Windows updater foundation
|
|
are implemented and docker-test/runtime-proven: backend exposes
|
|
`/mesh/v1/health` through the admin/nginx control-plane origin and issues
|
|
control-plane-only bootstrap rendezvous leases for outbound-only nodes using
|
|
their reported public control-plane URL. Remote Windows node
|
|
`ifcm-rufms-s-mo1cr` resolved 3/3 peers to `relay_ready` through
|
|
`http://94.141.118.222:19191`, while service/RDP/VPN payload forwarding
|
|
remains disabled. Release `0.1.3` is published for Docker and Windows
|
|
`windows_service` artifacts, and `install-windows` now installs a
|
|
per-node Scheduled Task updater for future Windows node-agent updates.
|
|
- C17Z22 updater observability and Windows host-agent self-update staging are
|
|
implemented and test-proven: `rap-host-agent` reports `phase=plan`,
|
|
`status=noop` for already-current/no-op plans, update state is scoped per
|
|
product so `rap-node-agent` and `rap-host-agent` do not overwrite each
|
|
other's current version, and the Windows updater wrapper runs short
|
|
one-shot cycles that can apply staged `rap-host-agent.exe.next` before the
|
|
next update check. Release `rap-host-agent 0.1.3` is published for
|
|
`linux_binary` and `windows_binary`; Docker updater containers on
|
|
`test-1/2/3` report no-op plans.
|
|
- Installation Authority foundation is implemented: production requires strict
|
|
Product Root public key config, first-owner bootstrap uses signed Ed25519
|
|
activation manifests, `installation_authority` and signed
|
|
`platform_role_grants` are persisted, and strict platform-admin checks ignore
|
|
direct `users.platform_role` database edits without a valid signed grant.
|
|
Web-admin exposes installation status/first-owner bootstrap, and
|
|
`scripts/installation/product-root-tool.go` generates keys/manifests for
|
|
offline product-root operations.
|
|
- Cluster Authority and node enrollment bootstrap are docker-test lifecycle
|
|
smoke-proven in run `dev-bootstrap-20260428-201430`: a fresh dev install
|
|
bootstrapped the first owner, created a cluster, issued a signed join token,
|
|
accepted real `rap-node-agent` enrollment, owner-approved the join request,
|
|
agent-polled signed bootstrap, persisted cluster authority pin, heartbeated,
|
|
and verified signed `c17z18.synthetic.v1` Control Plane config. Production
|
|
service payload forwarding remains unavailable.
|
|
- Migration `000021_cluster_authority_keys` drops/recreates
|
|
`cluster_admin_summaries` because fresh replay proved PostgreSQL cannot
|
|
change that view layout via `CREATE OR REPLACE VIEW`.
|
|
- `rap-node-agent` desired-workload polling/status reporting is gated by
|
|
`RAP_WORKLOAD_SUPERVISION_ENABLED=false` by default while service runtime
|
|
supervision remains a stub.
|
|
- C18 VPN/IP tunnel service target design is completed as documentation only.
|
|
- C18A VPN/IP tunnel control-plane data model foundation is implemented and
|
|
backend-test-proven.
|
|
- C18B VPN/IP tunnel lease/fencing hardening is implemented and
|
|
backend-test-proven.
|
|
- C18C VPN/IP tunnel node-agent desired-state consumption/reporting is
|
|
implemented and backend-test-proven.
|
|
- No next platform-core implementation step is automatically authorized after
|
|
C17Z20. The next mesh layer should stay limited to route-health feedback
|
|
refresh dampening/no-change cooldown unless the user explicitly chooses
|
|
another staged task.
|
|
- Latest RDP performance reference image:
|
|
`rap-rdp-worker:rdp-perf6-dirty-region`
|
|
- Stage 5.2 file-download runtime artifacts remain preserved for when RDP work
|
|
resumes, but they are not the active next task.
|
|
- Do not use `docker.cin.su` for this project unless explicitly requested for a separate one-off check.
|
|
|
|
### Backend
|
|
- Go
|
|
- PostgreSQL = source of truth
|
|
- Redis = live coordination / routing only
|
|
- REST for control plane
|
|
- WebSocket for live session channel
|
|
|
|
### Worker
|
|
- C++ worker
|
|
- FreeRDP integration
|
|
- worker runtime hides FreeRDP details from backend
|
|
- The C++ worker remains the primary RDP runtime.
|
|
- Target RDP performance direction: `docs/architecture/RDP_SERVICE_CPP_PERFORMANCE_TARGET.md`.
|
|
- The RDP performance rewrite scope is limited to C++ RDP service adapter
|
|
internals. It must not redesign backend control plane, cluster transport,
|
|
organizations, leases, or session lifecycle.
|
|
- The C# RDP service skeleton is inactive research scaffolding and is not the
|
|
current runtime direction.
|
|
- Current RDP Adapter baseline: RDP-Perf-6 dirty-region direct binary rendering
|
|
is completed and smoke-proven on `docker-test`. RDP work is paused by product
|
|
decision; next active work is Fabric Core / cluster foundation.
|
|
- P3/P3.1 security-readiness foundation exists: production mode rejects
|
|
plaintext credential-like resource metadata, requires `secret_ref` for
|
|
RDP/VNC/SSH resources, and has an encrypted PostgreSQL-backed resource secret
|
|
storage/resolver MVP. P3.2 direct-worker TLS/PKI guard exists.
|
|
- P3.3 production-like test-stand smoke is complete on `docker-test`: backend
|
|
runs in `APP_ENV=production` with a test-only secret key file, a secret-backed
|
|
RDP resource starts real sessions through the resolver path, metadata/audit do
|
|
not contain plaintext credentials, and backend gateway fallback remains
|
|
available when direct worker WSS trust is `smoke_insecure`.
|
|
- P3.4 production direct-worker WSS trust model is documented in
|
|
`docs/architecture/PRODUCTION_DIRECT_WORKER_WSS_TRUST.md`; it defines
|
|
platform CA/public CA behavior, worker certificate SAN/identity requirements,
|
|
app-local Windows trust direction, rotation/revocation, and the future
|
|
`platform_ca` smoke plan. No RDP runtime behavior changed in P3.4.
|
|
- P3.5 app-local platform CA trust is implemented and runtime-proven on
|
|
`docker-test`: Windows client validates direct worker WSS with an app-local
|
|
platform CA bundle, keeps hostname/SAN validation enabled, selects
|
|
`direct_worker_wss` without insecure TLS bypass, and falls back to backend
|
|
gateway for unknown CA / smoke-only production cases.
|
|
- P3.6 stale Redis worker/live event idempotency is implemented and
|
|
runtime-proven: stale worker events for terminal PostgreSQL sessions are
|
|
ignored, backend restart survives stale Redis events, and terminal sessions
|
|
are not reopened.
|
|
- Stage 5.2 server-to-client file download core data path is runtime-proven:
|
|
direct worker WSS and backend gateway fallback both download text/binary
|
|
files from `RAP_Transfers\ToClient` with matching size/hash, and direct
|
|
policy blocking is proven for `disabled` and `client_to_server`. Lifecycle
|
|
blocking is also runtime-proven for detach, old-client takeover, and worker
|
|
failure. Runtime report:
|
|
`artifacts/stage5-2-file-download-runtime-report.md`.
|
|
- Stage 5.2 is not fully accepted yet. Remaining proof: Windows desktop UI
|
|
download path and regression matrix for rendering/input/clipboard/upload/
|
|
reconnect/takeover.
|
|
|
|
### Clients
|
|
- future native clients:
|
|
- Windows: native desktop client first
|
|
- Linux: native desktop client later
|
|
- web UI is admin/control plane, not the primary power-user client
|
|
|
|
## Final architecture direction
|
|
|
|
The long-term target architecture is documented in:
|
|
|
|
- `docs/architecture/SECURE_ACCESS_FABRIC_TARGET.md`
|
|
- `docs/architecture/CLUSTER_NODE_ADMIN_FOUNDATION.md`
|
|
- `docs/architecture/WEB_INGRESS_AND_ADMIN_UI_MODEL.md`
|
|
|
|
This document defines the target Secure Access Fabric architecture only. It is not the current implementation scope and must not be used as permission to start mesh, VPN, multi-cluster, updater, or realtime data-plane migration work without an explicit staged prompt.
|
|
|
|
`CLUSTER_NODE_ADMIN_FOUNDATION.md` defines the next platform-core planning
|
|
baseline for clusters, node enrollment, native node-agent identity, platform
|
|
admin console, multi-cluster administration, and future organization admin
|
|
visibility. It is a staged foundation document, not permission to implement
|
|
mesh packet routing or VPN runtime.
|
|
|
|
`WEB_INGRESS_AND_ADMIN_UI_MODEL.md` defines WEB as HTTP/HTTPS ingress and
|
|
Admin UI presentation only. Cluster configuration remains Control Plane
|
|
ownership through scoped APIs, PostgreSQL source-of-truth mutations, and audit.
|
|
Dynamic pages must be safe schema-driven projections and must not embed
|
|
internal topology, peer caches, route caches, secrets, raw credentials, or
|
|
arbitrary executable code.
|
|
|
|
Admin endpoint placement is explicit. Fabric Storage / Config Storage nodes do
|
|
not automatically host or move the cluster panel. Platform Owner Console
|
|
remains global platform-owner scope. Cluster Admin Endpoint requires explicit
|
|
admin/web ingress role assignment, cluster health/trust readiness, and Control
|
|
Plane authorization. Organization Admin Panel remains a tenant-safe projection.
|
|
|
|
The final platform must support:
|
|
|
|
1. Multi-tenancy / Organizations
|
|
- platform has many organizations
|
|
- each organization has isolated users, groups, resources, policies, audit, connectors
|
|
- users may belong to multiple organizations
|
|
- organization admins only see their organization
|
|
- platform admins see platform scope
|
|
|
|
2. Identity federation
|
|
- local users
|
|
- LDAP / Active Directory
|
|
- OIDC
|
|
- future extensibility for more identity sources
|
|
- access mappings based on external groups / claims
|
|
|
|
3. Cluster of nodes
|
|
- no mandatory single central node
|
|
- many nodes across many sites
|
|
- nodes can be platform-managed or customer-managed
|
|
- customer-managed nodes are sandboxed cluster participants, not full cluster owners
|
|
|
|
4. Node agent
|
|
- small stable always-running agent on every node
|
|
- supervises services
|
|
- downloads updates
|
|
- verifies signed artifacts
|
|
- can rollback to previous version
|
|
- can restart crashed services
|
|
- can work on thin or thick nodes
|
|
|
|
5. Service-based node model
|
|
Each node is not monolithic.
|
|
A node has:
|
|
- capabilities: what it can do physically/technically
|
|
- enabled services: what it is allowed/assigned to do
|
|
|
|
Possible services include:
|
|
- ingress-gateway
|
|
- mesh-router
|
|
- relay
|
|
- connector-host
|
|
- vpn-adapter
|
|
- session-worker
|
|
- media-relay
|
|
- file-relay
|
|
- update-cache
|
|
- config-replica
|
|
- audit-sink
|
|
- metrics-exporter
|
|
|
|
6. Cluster mesh and routing
|
|
- encrypted inter-node communication
|
|
- dynamic topology
|
|
- no need for full mesh
|
|
- multi-hop routing allowed
|
|
- route failover
|
|
- client failover between ingress nodes
|
|
- connector failover between nodes
|
|
|
|
7. Split-brain prevention
|
|
- quorum-based cluster behavior
|
|
- minority partition must not become a second authoritative cluster
|
|
- degraded / recovery / isolated modes
|
|
- manual recovery / promote decision by platform recovery admin
|
|
|
|
8. Connector / VPN layer
|
|
- connectors are reusable network access methods
|
|
- one connector may be used by multiple resources
|
|
- connector placement and failover are controlled by policy
|
|
- nodes may be allowed or disallowed to host connectors
|
|
- direct access, VPN, relay and future egress modes must fit this model
|
|
|
|
9. Future exit mode
|
|
- split tunnel
|
|
- full tunnel
|
|
- internet access through cluster
|
|
- not first implementation priority
|
|
|
|
## Non-negotiable design rules
|
|
|
|
- Do not rewrite proven session lifecycle carelessly.
|
|
- Do not turn Redis into a source of truth.
|
|
- Do not make certificate-ignore a global worker setting.
|
|
- Do not make customer-managed nodes platform-wide trusted by default.
|
|
- Do not create a separate cluster per organization.
|
|
- Do not assume a single permanently reachable central node.
|
|
- Do not rely on “secret protocol with no docs” as security.
|
|
- Security must come from crypto, auth, isolation, policy and observability.
|
|
- Prefer incremental evolution from current proven system.
|
|
- Do not collapse platform control plane and data plane into one vague layer.
|
|
|
|
## Implementation strategy
|
|
|
|
The codebase must evolve in phases.
|
|
|
|
Current implementation focus remains:
|
|
- RDP work is paused by product decision
|
|
- preserve the accepted RDP Adapter baseline and Stage 5.x file-transfer work
|
|
- do not delete or rewrite the current RDP MVP while platform-core work starts
|
|
- C1-C9 platform-core foundations are implemented and verified: clusters,
|
|
node enrollment, node-agent scaffold, platform admin console, workload
|
|
supervision contract, mesh control-plane prep, mesh skeleton, multi-cluster
|
|
hardening, and organization admin foundation
|
|
- C10 Fabric Core configuration distribution design is completed
|
|
- C11 signed scoped cluster snapshot model is completed
|
|
- C12 node local state store is completed
|
|
- C13 Fabric Storage / Config Storage service foundation is completed
|
|
- C14 peer directory and cache model is completed
|
|
- C15 Fabric Routing Engine skeleton is completed
|
|
- C16 secure node-to-node channel lifecycle is completed
|
|
- C17 mesh routing runtime implementation plan is completed
|
|
- C17A synthetic mesh runtime skeleton is implemented and test-proven with
|
|
synthetic fabric messages only, no RDP/VPN/production service traffic
|
|
- C17B route health and failover probes are implemented and test-proven with
|
|
synthetic traffic only, no RDP/VPN/production service traffic
|
|
- C17C relay semantic hardening is implemented and test-proven with synthetic
|
|
channel classes only, no RDP/VPN/production service traffic
|
|
- C17D non-production test-service path is implemented and test-proven with
|
|
bounded `synthetic.echo` traffic only, no RDP/VPN/production service traffic
|
|
- C17E live node-to-node synthetic HTTP transport is implemented and
|
|
smoke-proven with synthetic traffic only
|
|
- C17F scoped synthetic route config loading and route-health reporting is
|
|
implemented and smoke-proven with synthetic traffic only
|
|
- C17G Control Plane scoped synthetic config read/consume is implemented and
|
|
test-proven with synthetic traffic only
|
|
- C17H deployed multi-agent synthetic config smoke is implemented and
|
|
runtime-proven on `docker-test` with synthetic traffic only
|
|
- C17I production forwarding gate foundation is implemented and test-proven;
|
|
production forwarding remains unavailable
|
|
- C17J production envelope contract validation is implemented and test-proven;
|
|
production forwarding remains unavailable
|
|
- C17K production envelope observation is implemented and test-proven;
|
|
production forwarding remains unavailable
|
|
- C17L bounded production observation sink is implemented and test-proven;
|
|
production forwarding remains unavailable
|
|
- C17M production observation sink wiring is implemented and test-proven;
|
|
production forwarding remains unavailable
|
|
- C17N production observation sink metrics are implemented and test-proven;
|
|
production forwarding remains unavailable
|
|
- C17O production observation sink local metrics logging is implemented and
|
|
test-proven; production forwarding remains unavailable
|
|
- C17P production observation sink change-driven metrics logging is implemented
|
|
and test-proven; production forwarding remains unavailable
|
|
- C17Q production forwarding gate/runtime log boundary is implemented and
|
|
test-proven; production forwarding remains unavailable
|
|
- C17R production observation sink capacity guard is implemented and
|
|
test-proven; production forwarding remains unavailable
|
|
- C17S production observation panic fail-closed hardening is implemented and
|
|
test-proven; production forwarding remains unavailable
|
|
- C17T production envelope payload boundary is implemented and test-proven;
|
|
production forwarding remains unavailable
|
|
- C17U production envelope created-at skew boundary is implemented and
|
|
test-proven; production forwarding remains unavailable
|
|
- C17V peer endpoint candidate model and NAT/connectivity hints are
|
|
implemented and test-proven; production forwarding remains unavailable
|
|
- C17W peer endpoint candidate scoring model is implemented and test-proven;
|
|
production forwarding remains unavailable
|
|
- C17X health-aware endpoint candidate scoring overlay is implemented and
|
|
test-proven; production forwarding remains unavailable
|
|
- C17Y Platform Owner synthetic mesh visibility is implemented and
|
|
build/test-proven; production forwarding remains unavailable
|
|
- C17Z production fabric-control direct forwarding is implemented and
|
|
test-proven; production service traffic remains unavailable
|
|
- C17Z1 production fabric-control multi-hop route-path forwarding is
|
|
implemented and test-proven; production service traffic remains unavailable
|
|
- C17Z2 production fabric-control forwarding observability is implemented and
|
|
test-proven; production service traffic remains unavailable
|
|
- C17Z3 production fabric-control route-config boundary is implemented and
|
|
test-proven; production service traffic remains unavailable
|
|
- C17Z4 scoped peer directory/recovery seed boundary is implemented and
|
|
test/build-proven; production service traffic remains unavailable
|
|
- C17Z5 node-agent peer cache runtime boundary is implemented and test-proven;
|
|
production service traffic remains unavailable
|
|
- C17Z6 dynamic endpoint reporting boundary is implemented and test-proven;
|
|
production service traffic remains unavailable
|
|
- C17Z7 private/corporate endpoint candidate boundary is implemented and
|
|
test-proven; production service traffic remains unavailable
|
|
- C17Z8 peer connection state machine boundary is implemented and test-proven;
|
|
production service traffic remains unavailable
|
|
- C17Z9 peer recovery planner boundary is implemented and test-proven;
|
|
production service traffic remains unavailable
|
|
- C17Z10 peer connection intent planner boundary is implemented and
|
|
test-proven; production service traffic remains unavailable
|
|
- C17Z11 peer connection manager runtime boundary is implemented and
|
|
test-proven; production service traffic remains unavailable
|
|
- C17Z12 rendezvous/relay control-plane contract is implemented and
|
|
docker-test-runtime-proven; production service traffic remains unavailable
|
|
- C17Z13 rendezvous lease telemetry is implemented and
|
|
docker-test-runtime-proven; production service traffic remains unavailable
|
|
- C17Z14 rendezvous lease refresh contract is implemented and
|
|
docker-test-runtime-proven; production service traffic remains unavailable
|
|
- C17Z15 backend relay replacement policy is implemented and
|
|
docker-test-runtime-proven; production service traffic remains unavailable
|
|
- C17Z16 route/path decision artifact is implemented and
|
|
docker-test-runtime-proven; production service traffic remains unavailable
|
|
- C17Z17 node-side route generation tracker is implemented and
|
|
docker-test-runtime-proven; production service traffic remains unavailable
|
|
- C17Z18 synthetic route-health effective path runtime is implemented and
|
|
docker-test-runtime-proven; production service traffic remains unavailable
|
|
- C17Z19 synthetic route-health feedback scoring is implemented and
|
|
docker-test-runtime-proven; production service traffic remains unavailable
|
|
- C17Z20 node-side route-health feedback refresh is implemented and
|
|
docker-test-runtime-proven; production service traffic remains unavailable
|
|
- C17Z21 node installation/update control-plane is implemented and
|
|
docker-test-runtime-proven for Docker nodes; production service traffic
|
|
remains unavailable
|
|
- C17Z22 Windows host-agent install/update supervision is implemented and
|
|
runtime-proven on the remote Windows node; production service traffic remains
|
|
unavailable
|
|
- C17Z23 update observability is implemented in backend/admin UI: per-node
|
|
updater status history is exposed and deployed on docker-test, so node-agent
|
|
and host-agent update activity can be audited from node details
|
|
- C17Z24 combined updater reporting is implemented and docker-test-proven:
|
|
Linux/Docker `rap-host-agent update-loop` now also polls/reports
|
|
`rap-host-agent` status, release `0.1.4` is published for node-agent and
|
|
host-agent artifacts, and docker-test nodes `test-1/2/3` auto-updated to
|
|
node-agent `0.1.4` while reporting host-agent `0.1.4` no-op status.
|
|
- C17Z25 Windows updater repair visibility is implemented in admin UI: node
|
|
details / Updates now shows a ready CMD repair command for existing Windows
|
|
nodes using `http://vpn.cin.su:19191/api/v1`, `--replace`, and
|
|
`--auto-update-current-version 0.0.0` so a stale updater wrapper can be
|
|
recreated without a new join token.
|
|
- C17Z26 updater fleet visibility is implemented in admin UI: the node list now
|
|
shows per-node updater status based on latest `rap-node-agent` and
|
|
`rap-host-agent` reports, explicitly flagging missing host-agent reports,
|
|
stale update reports, or update errors before opening node details.
|
|
- C17Z27 backend version-state projection is implemented and deployed on
|
|
docker-test: node list responses now derive `version_state` from active
|
|
`rap-node-agent` desired policy plus latest update report. Docker/Linux nodes
|
|
on `0.1.4` show `current`; the remote Windows node still on `0.1.3` shows
|
|
`outdated` while remaining heartbeat-healthy.
|
|
- C17Z28 Windows updater loop hardening is implemented and partially
|
|
docker-test-proven via release `0.1.5`: Windows host-agent updater scripts now
|
|
run combined `update-loop --max-runs 1`, and Windows `update-loop` also
|
|
polls/applies `rap-host-agent` updates. Release `0.1.5` artifacts are
|
|
published for Docker/Linux and Windows; docker-test nodes `test-1/2/3`
|
|
updated to `rap-node-agent 0.1.5`. Existing remote Windows nodes with stale
|
|
pre-0.1.5 updater wrapper still require one repair command from admin UI to
|
|
replace their local wrapper, after which automatic polling should continue.
|
|
- Admin UI now marks missing host-agent updater reports as `repair updater` in
|
|
the node list and explains in node details / Updates when to run the Windows
|
|
repair command. The command uses the external control-plane endpoint and does
|
|
not require a join token for already enrolled Windows nodes.
|
|
- Admin UI node details / Updates also provides a ready downloadable
|
|
`rap-repair-updater-<node>.cmd` plus copy-command action for Windows repair,
|
|
reducing operator copy/paste mistakes on remote Windows hosts.
|
|
- Windows repair command generation was hardened after the first remote repair:
|
|
foreground `update-loop` now includes explicit `--node-id`, copies any staged
|
|
`rap-host-agent.exe.next` over the main host-agent binary after the one-shot
|
|
loop exits, deletes the staged file, and runs the updater scheduled task.
|
|
The node list now distinguishes `host-agent staged` from generic stale/error.
|
|
- C17Z29 Windows persistent updater repair is implemented in `rap-host-agent`
|
|
release `0.1.6`: `install-windows` accepts `--node-id` and writes that node
|
|
id into the persistent Windows updater wrapper so Scheduled Task polling no
|
|
longer depends on finding `identity.json` in the expected state directory.
|
|
Docker-test nodes `test-1/2/3` updated to `0.1.6`; existing Windows and
|
|
off-host Docker nodes still need their local updater wrappers to pick up the
|
|
0.1.6 host-agent repair path.
|
|
- C17Z30 operator-configured public mesh endpoints are implemented and
|
|
docker-test-deployed: desired `mesh-listener.advertise_endpoint` is now
|
|
projected into peer endpoint candidates for other nodes and preferred over
|
|
auto-discovered private heartbeat endpoints. `home-1`
|
|
(`8ad04829-cd30-4290-913d-1ce5c7ef7bb3`) is configured with
|
|
`listen_addr=0.0.0.0:19131`, `advertise_endpoint=http://94.141.118.222:19199`,
|
|
`connectivity_mode=direct`, `nat_type=port_restricted`, `region=home`.
|
|
`test-1` synthetic config now receives `home-1` peer endpoint
|
|
`http://94.141.118.222:19199`; internal `192.168.200.85:19131` responds with
|
|
HTTP 405 on GET, while external `94.141.118.222:19199` currently refuses TCP,
|
|
so router/firewall forwarding still needs correction outside the platform.
|
|
- C17Z31 offsite bootstrap peer selection is implemented and docker-test
|
|
deployed: operator-configured public/direct desired mesh-listener endpoints
|
|
are kept in core-mesh bootstrap even after the default warm-peer target is
|
|
reached. This fixes the case where remote Windows node
|
|
`ifcm-rufms-s-mo1cr` received only `test-*` warm peers and no `home-1`.
|
|
Its synthetic config now includes `home-1` endpoint
|
|
`http://94.141.118.222:19199` and candidates ordered as operator public,
|
|
heartbeat advertised public, then private LAN converted to relay-required for
|
|
offsite. External TCP to `94.141.118.222:19199` still failed from Codex and
|
|
docker-test checks while internal `192.168.200.85:19131` succeeds, so a real
|
|
offsite `Test-NetConnection 94.141.118.222 -Port 19199` is the next network
|
|
validation.
|
|
- C17Z32 native Ubuntu/Linux service install is implemented and docker-test
|
|
deployed: backend exposes `/node-agents/linux-install-profile`, host-agent
|
|
supports `install-linux`, installs `rap-node-agent` under
|
|
`/opt/rap/<node>`, state under `/var/lib/rap/nodes/<node>`, config under
|
|
`/etc/rap/<node>`, creates `rap-node-agent-<node>.service`, and creates a
|
|
persistent `rap-host-agent-updater-<node>.service` for automatic node-agent
|
|
and host-agent updates. Release `0.1.7` is published for `rap-node-agent`
|
|
(`linux_binary`, `windows_service`) and `rap-host-agent`
|
|
(`linux_binary`, `windows_binary`). Admin UI now has an `Ubuntu service`
|
|
install profile and generates profile-based `install-linux` commands.
|
|
A one-use token for `vps-ubuntu-1` is active until 2026-05-02T08:41:41Z:
|
|
`rap_join_a23Xhz63YstshWUBAPGPz5fzQ8YpHDP05RXaaYa4DoA`; scope roles are
|
|
`core-mesh` and `relay-node`, control-plane endpoint is
|
|
`http://vpn.cin.su:19191/api/v1`, artifact endpoint is
|
|
`http://vpn.cin.su:19191/downloads`.
|
|
- Admin UI and docs now cover the full Windows updater operational workflow:
|
|
node details shows an `Updater health` summary, generated repair CMD prints
|
|
scheduled-task and binary diagnostics before/after repair, applies staged
|
|
host-agent binaries, restarts the updater task, and README documents first
|
|
install, repair without join-token, system-task/user-task behavior, staged
|
|
host-agent recovery, and reboot/autostart verification.
|
|
- Cluster Authority plus node enrollment bootstrap polling are docker-test
|
|
lifecycle-smoke-proven; fresh install migration replay is fixed for
|
|
`cluster_admin_summaries`
|
|
- C18 VPN/IP tunnel service target design is completed as documentation only
|
|
- C18A VPN/IP tunnel control-plane data model foundation is implemented and
|
|
backend-test-proven
|
|
- C18B VPN/IP tunnel lease/fencing hardening is implemented and
|
|
backend-test-proven
|
|
- C18C VPN/IP tunnel node-agent desired-state consumption/reporting is
|
|
implemented and backend-test-proven
|
|
- Version Storage / Update Repository is documented as a future Fabric Core
|
|
service for signed release manifests, OS/arch artifacts,
|
|
stable/current/candidate channels, update-cache mirroring, node-agent
|
|
update supervision, rollback, and explicit data-structure migration bundles.
|
|
Runtime updater behavior is partially implemented for the current Docker and
|
|
Windows node-agent/host-agent paths; broader staged rollout policy and
|
|
service payload forwarding remain separate work.
|
|
- no next platform-core implementation step is automatically authorized after
|
|
C17Z20; choose the next narrow staged prompt explicitly before continuing
|
|
- preserve the proven RDP lifecycle behavior
|
|
- keep the current backend gateway available as the active/fallback implementation path
|
|
- accepted VPN data-plane target: the phone/client connects only to an
|
|
available entry node; the entry node uses the existing mesh/fabric route to a
|
|
selected exit node/pool, and the exit node handles LAN/internet egress. Nodes
|
|
behind NAT may participate when they can maintain outbound mesh/control
|
|
sessions. Backend packet relay must remain a compatibility/fallback path, not
|
|
the desired steady-state path.
|
|
- C18D VPN-over-fabric foundation is implemented and docker-test-started:
|
|
VPN client profiles include `vpn_fabric_route` with entry pool, exit pool,
|
|
selected entry/exit, preferred `fabric_mesh` data-plane, and
|
|
`backend_relay` fallback. Node-agent `0.2.39` adds a dedicated production
|
|
`vpn_packet` channel (`vpn.packet_batch`, 256 KiB batch limit), destination
|
|
delivery hook, `vpnruntime.FabricPacketTransport`, and
|
|
`vpn_fabric_packet_transport` heartbeat capability. `home-1` auto-updated to
|
|
`0.2.39`; other nodes have automatic desired policy `0.2.39` and should move
|
|
as their updater loops pick it up. Live Android VPN traffic still uses backend
|
|
relay until entry-node client ingress is wired to the fabric transport.
|
|
- C18E VPN-over-fabric route contract is backend-deployed on docker-test as
|
|
`rap-backend:test-vpn-fabric-route-0.2.41`: when a VPN client profile selects
|
|
different entry and exit nodes, backend now ensures two active
|
|
`mesh_route_intents` with service_class `vpn_packets` and allowed channel
|
|
`vpn_packet`. The live HOME profile currently selects `usa-los-1` as entry
|
|
and `home-1` as exit when `entry_node_id=b829ffde-...` is requested, and the
|
|
synthetic config for both nodes includes the two `vpn_packet` routes. Existing
|
|
fallback remains `backend_relay`; production forwarding gate is still disabled
|
|
on old/live remote nodes until their runtime is explicitly updated/enabled.
|
|
- External/offsite updater gap found and fixed for version `0.2.40`: native
|
|
`rap-node-agent` binaries for `linux_binary`, `linux_service`, and
|
|
`windows_service` plus matching `rap-host-agent` binaries are copied under
|
|
`/downloads` and registered in channel `dev-external`. Update plans for
|
|
`usa-los-1` (`linux_binary`) and `ifcm-rufms-s-mo1cr` (`windows_service`) now
|
|
return `action=update`, `target_version=0.2.40` instead of
|
|
`no_matching_artifact`.
|
|
- C18F production-forwarding gate work is partially live: backend
|
|
`rap-backend:test-vpn-fabric-route-0.2.42` signs node synthetic configs with
|
|
`production_forwarding=true` / `control_plane_only=false` when the node's
|
|
desired `mesh-listener` workload has `production_forwarding_enabled=true`.
|
|
`home-1` and `usa-los-1` desired mesh-listener configs have this flag enabled.
|
|
Node-agent `0.2.44` accepts signed production-forwarding mesh configs and
|
|
host-agent `0.2.44` fixes Docker updater behavior so synthetic mesh runtime is
|
|
not disabled on Docker updates. Runtime status: `usa-los-1` reports
|
|
`mesh_production_forwarding=true`; `home-1` reports `0.2.44` and synthetic
|
|
runtime enabled, but its listener report is still `disabled/listen_addr_empty`,
|
|
so `home-1` is not yet a usable production fabric endpoint. Next action is to
|
|
repair why `home-1` is not applying the signed mesh-listener config
|
|
(`listen_addr=0.0.0.0:19131`) after Docker updater restart.
|
|
- C18G VPN-over-fabric runtime path is live-tested on docker-test. Backend is
|
|
deployed as `rap-backend:test-vpn-fabric-route-0.2.43`; VPN route intents now
|
|
allow both `vpn_packet` data and `fabric_control` health probes. Node-agent
|
|
`0.2.47` fixes initial production VPN packet envelope hop addressing and
|
|
reports the matching version. `home-1` and `usa-los-1` both report
|
|
`0.2.47`, healthy, listener `0.0.0.0:19131`, and
|
|
`mesh_production_forwarding=true`. Live route health is reachable in both
|
|
directions (`usa-los-1 -> home-1` around 200 ms, `home-1 -> usa-los-1`
|
|
around 200-415 ms). A direct live POST to
|
|
`http://195.123.240.88:19131/api/v1/clusters/.../vpn-connections/.../tunnel/client/packets`
|
|
returns `202 Accepted`, proving entry-node VPN packet ingress can forward
|
|
over fabric to the home exit. The HOME VPN placement policy now has entry
|
|
pool `[usa-los-1, home-1]` and exit `home-1`; client profile with preferred
|
|
`usa-los-1` selects `usa-los-1 -> home-1`.
|
|
- C18H live VPN triage on 2026-05-04: `home-1` and `usa-los-1` report
|
|
node-agent `0.2.48`, healthy heartbeats, active HOME VPN assignment on
|
|
`home-1`, and `packet_forwarding=true` / `runtime_available=true`. Manual
|
|
packet tests through the USA entry proved the path
|
|
Android-style packet -> `usa-los-1` -> fabric -> `home-1` -> LAN/DNS ->
|
|
fabric -> `usa-los-1` -> client can return ICMP and DNS replies. The remaining
|
|
live symptom was the phone not sending fresh packets to the current entry
|
|
after the backend relay queue was cleared. Android VPN app `0.2.59` was built
|
|
and published to `/downloads/rap-android-rdp-vpn-latest-debug.apk`; it
|
|
normalizes old saved backend URLs (`vpn.cin.su:19191`,
|
|
`94.141.118.222:19191`, `192.168.200.61:18080`, etc.) to the current USA
|
|
entry backend `http://195.123.240.88:19131/api/v1` and shows app version,
|
|
device id, and connection id in the header for live log correlation.
|
|
- C18I fabric service-channel foundation is live on 2026-05-07. Backend,
|
|
node-agent, and Android VPN release `0.2.159` are published. VPN profiles now
|
|
include a signed `rap.fabric_service_channel_lease.v1` with
|
|
`entry_direct_http_v1` packet and WebSocket templates. Android consumes this
|
|
lease and sends service-channel headers. The `usa-los-1` entry endpoint
|
|
validates the cluster-authority signed lease payload and token hash; a live
|
|
smoke through `http://195.123.240.88:19131/.../fabric/service-channels/...`
|
|
succeeded with a valid lease and rejected a bad token with `403`. Current HOME
|
|
profile selects `usa-los-1` as entry and `home-1` as exit; both nodes report
|
|
`0.2.159`. Docker-test nodes `test-1`, `test-2`, and `test-3` also report
|
|
`0.2.159`. `ifcm-rufms-s-mo1cr` is still on `0.2.119`; it has staged the
|
|
host-agent `0.2.159` update and should finish on the next Windows updater
|
|
loop/restart.
|
|
- C18J fabric service-channel runtime route-manager slice is live on
|
|
2026-05-07 as node/host-agent `0.2.162`. The entry-node
|
|
`FabricClientPacketIngress` now preserves its runtime object across synthetic
|
|
config refreshes, so heartbeat telemetry reports the same ingress object that
|
|
serves HTTP/WebSocket service-channel traffic. It tracks send/receive batches,
|
|
route attempts/failures, selected route/next hop, local-gateway fallback, and
|
|
inbox queue depths. `SendClientPacketBatch` now retries all valid
|
|
`vpn_packet` route candidates with sticky preference before backend relay is
|
|
allowed as degraded compatibility fallback. Release `0.2.161` was superseded
|
|
because its Docker tar was rebuilt after registration; `0.2.162` is the
|
|
clean published release with matching artifact hashes. Docker-test
|
|
`test-1/2/3`, `usa-los-1`, and `ifcm-rufms-s-mo1cr` report `0.2.162`;
|
|
`home-1` is healthy and still on `0.2.161` awaiting its next updater loop.
|
|
Live smoke through `http://195.123.240.88:19131/.../fabric/service-channels`
|
|
returned `202` and `usa-los-1` telemetry then showed route attempts,
|
|
one route failure, and selected next hop `home-1`, proving live ingress
|
|
telemetry and alternate-route retry are active.
|
|
- C18K service-neutral flow/channel scheduler is live on 2026-05-07 as
|
|
node/host-agent `0.2.163`. The VPN proving service still carries universal
|
|
IP packets and does not route by application protocol, but the entry runtime
|
|
now hashes packets by IP 5-tuple, or packet hash for non-IP/invalid packets,
|
|
into 32 logical `flow-*` channels. Each channel has bounded queue accounting,
|
|
high-watermark/backpressure/dropped telemetry, and batches are fanned out per
|
|
logical channel before being sent through the same fabric route-manager. Live
|
|
smoke against `usa-los-1` posted two different IP flows through the signed
|
|
service-channel endpoint and heartbeat reported `send_packets=2`,
|
|
`send_flow_batches=2`, `flow_scheduler.channel_count=2`, `enqueued=2`,
|
|
`dequeued=2`, `dropped=0`, with queue depths for `flow-12` and `flow-14`.
|
|
All six current cluster nodes (`home-1`, `usa-los-1`, `ifcm-rufms-s-mo1cr`,
|
|
`test-1`, `test-2`, `test-3`) report node-agent `0.2.163` and healthy.
|
|
- C18L active flow scheduling telemetry is live on 2026-05-07 as
|
|
node/host-agent `0.2.164`. Each `flow-*` channel now keeps route memory,
|
|
served count, last served time, last route/next hop, failed-route marker,
|
|
consecutive failures, stall count, last send duration, and explicit
|
|
`route_rebuild_recommended` / `degraded_fallback_recommended` signals. The
|
|
scheduler drains non-stalled channels first, prefers less-served/older
|
|
channels, avoids a channel's last failed route on the next send, and only
|
|
marks degraded fallback after repeated failures. Live smoke against
|
|
`usa-los-1` posted two IP flows through the signed service-channel endpoint:
|
|
heartbeat reported schema `c18l.fabric_service_channel_runtime_report.v1`,
|
|
`send_packets=2`, `send_flow_batches=2`, `flow_scheduler.channel_count=2`,
|
|
`dropped=0`, `backpressure=false`, `last_next_hop=home-1`, and per-flow
|
|
`served=1`. One stale candidate route failed and was bypassed before the
|
|
successful route to `home-1`. All six current cluster nodes (`home-1`,
|
|
`usa-los-1`, `ifcm-rufms-s-mo1cr`, `test-1`, `test-2`, `test-3`) report
|
|
node-agent `0.2.164` and healthy.
|
|
- C18M Control Plane service-channel feedback is live on 2026-05-07. Backend
|
|
image `rap-backend:fabric-service-channel-0.2.165` is deployed on
|
|
docker-test, and node/host-agent `0.2.165` artifacts are published. When
|
|
issuing `rap.fabric_service_channel_lease.v1`, backend now reads fresh
|
|
entry-node heartbeat metadata
|
|
`fabric_service_channel_runtime_report.ingress.flow_scheduler.channel_stats`,
|
|
builds per-route service-channel feedback, boosts recently successful routes,
|
|
penalizes recent failures, and fences routes that report
|
|
`route_rebuild_recommended`, `degraded_fallback_recommended`, or repeated
|
|
consecutive failures. Fenced routes are not selected as primary or alternate;
|
|
if all selected entry/exit routes are fenced, the lease uses explicit
|
|
degraded backend fallback with reason
|
|
`fabric_routes_fenced_by_service_channel_feedback`. Live smoke created two
|
|
short-lived `test-1 -> test-2` route intents, injected a fresh
|
|
service-channel flow feedback heartbeat marking the higher-priority route as
|
|
rebuild-required, and the next lease selected the lower-priority healthy
|
|
route with score reason `service_channel_recent_success`; the bad route was
|
|
not offered as an alternate. Current node rollout: `home-1`, `usa-los-1`,
|
|
`test-1`, `test-2`, and `test-3` report `0.2.165`; Windows `ifcm-rufms-s-mo1cr`
|
|
remains healthy on `0.2.164` and should move on its next updater cycle.
|
|
- C18N durable service-channel route feedback is live on 2026-05-07. Backend
|
|
image `rap-backend:fabric-service-channel-0.2.166` is deployed on
|
|
docker-test with migration `000025_fabric_service_channel_route_feedback`.
|
|
Heartbeats now persist service-neutral route observations into
|
|
`fabric_service_channel_route_feedback_observations` and maintain an
|
|
expiring latest view in `fabric_service_channel_route_feedback_latest`.
|
|
Lease selection reads this durable latest feedback before falling back to
|
|
in-memory heartbeat parsing, so route fencing survives backend restarts and
|
|
stale heartbeat replacement. Node/host-agent `0.2.166` artifacts and Docker
|
|
image are published, update policies target `0.2.166`, and `test-1/2/3`,
|
|
`usa-los-1`, and `ifcm-rufms-s-mo1cr` report `0.2.166`; `home-1` is healthy
|
|
but still on `0.2.165` until its next updater cycle. Live smoke created two
|
|
short-lived `test-1 -> test-2` routes, persisted a fenced observation for the
|
|
higher-priority bad route and a healthy observation for the lower-priority
|
|
route, restarted backend, and the next lease selected the healthy route with
|
|
`service_channel_recent_success`.
|
|
- C18O service-channel feedback diagnostics and synthetic route avoidance are
|
|
live on 2026-05-07. Backend image
|
|
`rap-backend:fabric-service-channel-0.2.167` is deployed on docker-test and
|
|
web-admin is rebuilt/published. Admin/API now expose fresh durable feedback
|
|
through `GET /clusters/{clusterID}/fabric/service-channels/route-feedback`,
|
|
and each node synthetic config includes
|
|
`service_channel_route_feedback` with healthy/degraded/fenced counts and
|
|
observations. Synthetic config generation skips routes fenced by the local
|
|
node's durable service-channel feedback, so nodes stop receiving known-bad
|
|
route configs while the feedback is active. Live smoke created fresh
|
|
`test-1 -> test-2` routes, persisted `fenced` feedback for the higher-priority
|
|
route and `healthy` feedback for the lower-priority route, confirmed the API
|
|
returned both observations, and confirmed `test-1` synthetic config excluded
|
|
the bad route while keeping the healthy route.
|
|
- C18P proactive service-channel replacement decisions are live on 2026-05-07.
|
|
Backend image `rap-backend:fabric-service-channel-0.2.168` is deployed on
|
|
docker-test and web-admin is rebuilt/published. When synthetic config
|
|
generation withholds a route fenced by local service-channel feedback, it now
|
|
records a `route_path_decisions` item with
|
|
`decision_source=service_channel_feedback_replacement`,
|
|
`replacement_route_id`, effective replacement hops, and score reasons. If no
|
|
alternate exists, the decision source becomes
|
|
`service_channel_feedback_no_alternate` with visible score reason
|
|
`no_unfenced_alternate_route`. Live smoke created fresh `test-1 -> test-2`
|
|
bad/good routes, fenced the bad route, disabled older smoke routes, and
|
|
confirmed `test-1` synthetic config excluded the bad route, kept the good
|
|
route, and reported replacement from bad route to good route.
|
|
- C18Q service-channel replacement dampening is live on 2026-05-07. Backend
|
|
image `rap-backend:fabric-service-channel-0.2.169`, node/host-agent
|
|
`0.2.169` artifacts, Docker image, update policies, and web-admin are
|
|
published on docker-test. Replacement selection now gives a large stable
|
|
preference to routes with active healthy durable feedback, adding
|
|
`active_healthy_feedback_dampening_window` to score reasons, so a recently
|
|
successful replacement wins over a higher-priority but unproven route until
|
|
the feedback window expires or a newer fenced/healthy observation changes the
|
|
state. `RoutePathDecisionReport` now includes `degraded_decision_count` for
|
|
`service_channel_feedback_no_alternate`, and node-agent heartbeat reports
|
|
include `replacement_route_id` and degraded counts after upgrade. Live smoke
|
|
fenced a high-priority bad `test-1 -> test-2` route, supplied healthy feedback
|
|
for a low-priority route, also created a higher-priority unproven route, and
|
|
confirmed replacement selected the healthy route because of the dampening
|
|
window.
|
|
- C18Q hotfix `0.2.171` is published on 2026-05-07. Node-agent now includes
|
|
`service_channel_route_feedback` in the signed synthetic config model before
|
|
recalculating the authority payload hash. Without this, upgraded backend
|
|
configs were signed correctly but `0.2.169` agents rejected them with
|
|
`control-plane synthetic mesh config authority payload hash mismatch`.
|
|
Regression coverage verifies a signed config containing durable
|
|
service-channel feedback. Artifacts, Docker image, latest download aliases,
|
|
and update policies were moved to `0.2.171`; `test-1/2/3` are running
|
|
`0.2.171` and loading `source=control_plane` again. The release includes
|
|
`linux_service`, Docker, Windows service, and binary artifacts so service
|
|
installs can auto-update. Old C18 smoke/expired route intents were disabled
|
|
after validation.
|
|
- C18R fleet diagnostics/operator action slice is live on 2026-05-07. Backend
|
|
image `rap-backend:fabric-service-channel-0.2.172` adds route feedback
|
|
filters (`route_id`, `feedback_status`, `include_expired`) and
|
|
`POST /clusters/{clusterID}/fabric/service-channels/route-feedback/expire`.
|
|
The expire action is cluster-mutable/admin gated and marks latest feedback
|
|
expired without deleting historical observations. Web-admin / Fabric Links
|
|
now shows a cluster-level service-channel feedback panel with fenced,
|
|
degraded, healthy and no-alternate counts, replacement/no-alternate decisions,
|
|
and an operator `expire` action for stale non-healthy feedback.
|
|
- C18S service-channel feedback churn guardrails are implemented on
|
|
2026-05-07. Operator expire now records
|
|
`fabric.service_channel_route_feedback.expired` audit events, returns and
|
|
persists a short `operator_retry_cooldown_until`, and route generation adds
|
|
`service_channel_route_retry_after_operator_expire` when a manually expired
|
|
route is being retried. During that cooldown, repeated non-healthy feedback
|
|
from the same reporter/route/service is suppressed as
|
|
`operator_retry_cooldown` instead of immediately fencing the route again.
|
|
Web-admin shows the retry/cooldown state in Fabric Links.
|
|
- C18T automatic rebuild decision contract is implemented on 2026-05-07.
|
|
`RoutePathDecision` now carries `rebuild_request_id`, `rebuild_status`,
|
|
`rebuild_reason`, and `rebuild_attempt`. When fenced service-channel feedback
|
|
keeps failing outside manual retry cooldown, Control Plane records a bounded
|
|
rebuild request. If an unfenced alternate exists, the decision is marked
|
|
`rebuild_status=applied`; if not, it is
|
|
`pending_degraded_fallback` and leases expose backend relay with reason
|
|
`fabric_route_rebuild_pending_backend_relay`. Web-admin shows rebuild counts,
|
|
status, and attempts in Fabric Links. A live smoke on docker-test created
|
|
short-lived `test-1 -> test-2` bad/good routes, reported fenced feedback for
|
|
the bad route and healthy feedback for the good route, and confirmed scoped
|
|
synthetic config returned `service_channel_feedback_replacement` with
|
|
`rebuild_status=applied` and `rebuild_attempt=3`. Node/host-agent `0.2.175`
|
|
is published so agents preserve the new signed rebuild fields.
|
|
- C18U node-agent route-manager rebuild consumption is live on 2026-05-07.
|
|
Node-agent `0.2.176` now converts backend rebuild decisions into a
|
|
service-channel route-manager snapshot, counts rebuild requests/applies,
|
|
marks applied/pending-degraded routes as withdrawn, clears a withdrawn cached
|
|
selected route, and excludes withdrawn routes from new service-channel route
|
|
candidates. This keeps new flows from retrying a route that Control Plane has
|
|
already rebuilt away from. Unit coverage verifies a bad route is skipped in
|
|
favor of its replacement. Node/host-agent `0.2.176` artifacts, Docker image,
|
|
latest download aliases, release manifests, and node policies are published.
|
|
`test-1/2/3`, `usa-los-1`, and `ifcm-rufms-s-mo1cr` report `0.2.176`.
|
|
Backend `rap-backend:fabric-service-channel-0.2.176` is deployed with a
|
|
panel consistency fix: if a node reports the target version, stale failed
|
|
update status no longer overrides `version_state=current`.
|
|
- C18V route-manager churn telemetry is live on 2026-05-07. Node-agent
|
|
`0.2.177` adds `route_manager_transition` to the service-channel runtime
|
|
report with previous/current generation, transition status, decision counts,
|
|
withdrawn/restored route counts, pending-degraded fallback count, rebuild
|
|
applied count, and any cleared cached route. Tests cover applied rebuild
|
|
replacement, pending degraded fallback with no alternate, and restoration by
|
|
a fresh config so withdrawn routes do not become sticky local state. Artifacts,
|
|
Docker image, latest download aliases, release manifests, and node policies
|
|
are published. `test-1/2/3` run `0.2.177`; their heartbeat metadata exposes
|
|
`rap.fabric_service_channel_route_manager_transition.v1`.
|
|
- C18W live Control Plane/runtime verification is implemented and smoke-passed
|
|
on 2026-05-07. Script
|
|
`scripts/fabric/c18w-service-channel-route-manager-smoke.ps1` drives the
|
|
whole loop against docker-test API: creates temporary service-channel route
|
|
intents for `test-1 -> test-2`, injects fenced/healthy route feedback through
|
|
heartbeat, verifies scoped config emits `rebuild_status=applied`, waits for
|
|
node-agent heartbeat `route_manager_transition.status=applied_rebuild`,
|
|
expires the feedback, verifies the restored config has no rebuild decision,
|
|
and waits for `restored_by_new_config`. Result artifact:
|
|
`artifacts/c18w-service-channel-route-manager-smoke-result.json` with run
|
|
`c18w-20260507-173226`. During the smoke, operator expire exposed live pgx
|
|
parameter issues; backend `rap-backend:fabric-service-channel-0.2.179` is
|
|
deployed with safer UUID/text timestamp handling for feedback expire.
|
|
- C18X logical-channel isolation and bounded backpressure coverage is
|
|
implemented and smoke-passed on 2026-05-07. Node-agent/host-agent `0.2.180`
|
|
artifacts, Docker image, latest download aliases, release manifests, and
|
|
node policies are published. The key runtime fix is in
|
|
`FabricClientPacketIngress.routeCandidatesForChannel`: a channel with a local
|
|
failed-route avoid state no longer falls back to the global last selected
|
|
route, so one degraded logical flow cannot drag unrelated flows back onto the
|
|
failed path. Coverage proves independent logical-channel failover, bounded
|
|
same-channel backpressure/drop telemetry, and packet-flow hashing. Script
|
|
`scripts/fabric/c18x-service-channel-logical-channel-smoke.ps1` passes with
|
|
result artifact `artifacts/c18x-service-channel-logical-channel-smoke-result.json`
|
|
run `c18x-20260507-180647`. Test docker nodes `test-1/2/3` are running
|
|
`rap-node-agent:0.2.180`; backend remains
|
|
`rap-backend:fabric-service-channel-0.2.179`.
|
|
- C18Y route-intent lifecycle cleanup is implemented and smoke-passed on
|
|
2026-05-07. Backend `rap-backend:fabric-service-channel-0.2.181` is deployed
|
|
on docker-test, and web-admin Fabric Links now shows route-intent lifecycle
|
|
counts/table with operator `expire` and `disable` actions. Route intents are
|
|
enriched with `lifecycle_status`, `is_expired`, and `policy_expires_at`.
|
|
Node-scoped synthetic mesh config now filters out expired policy routes, so
|
|
stale smoke routes no longer get emitted to agents for route-health probing.
|
|
API actions are available at
|
|
`POST /clusters/{clusterID}/mesh/route-intents/{routeIntentID}/expire` and
|
|
`/disable`. Script `scripts/fabric/c18y-route-intent-lifecycle-smoke.ps1`
|
|
passed against docker-test API, result
|
|
`artifacts/c18y-route-intent-lifecycle-smoke-result.json` run
|
|
`c18y-20260507-192702`. During deploy, docker-test root disk was full from
|
|
build cache/images; `docker builder prune -af` and `docker image prune -f`
|
|
freed space before redeploy.
|
|
- C18Z bounded service-channel load coverage is implemented, published, and
|
|
smoke-passed on 2026-05-07. Node-agent/host-agent `0.2.181` artifacts,
|
|
Docker image `rap-node-agent:0.2.181`, latest download aliases, release
|
|
manifests, and update policies are published. `test-1/2/3` are restarted on
|
|
`rap-node-agent:0.2.181`; `usa-los-1` also reports `0.2.181`. The key runtime
|
|
fix is in `FabricFlowScheduler.Snapshot`: backpressure remains visible when
|
|
bounded drops occurred, even after the queue drains. Coverage proves
|
|
multi-channel rebuild away from a withdrawn primary route and per-channel
|
|
bounded drop/high-water telemetry. Script
|
|
`scripts/fabric/c18z-service-channel-load-smoke.ps1` passed against
|
|
docker-test API, result
|
|
`artifacts/c18z-service-channel-load-smoke-result.json` run
|
|
`c18z-20260507-194616`. Release artifacts were corrected after initial
|
|
publication to use backend-relative `/downloads/...` primary URLs plus
|
|
internal/external mirror URLs, so offsite nodes resolve downloads through
|
|
their own control-plane origin such as `http://vpn.cin.su:19191`. Current
|
|
caveat: `ifcm-rufms-s-mo1cr` and `home-1` remained `version_state=failed`
|
|
at the last check; their next update plan now points to reachable `0.2.181`
|
|
artifacts, but the local updater loop still needs to retry/report success.
|
|
- C18Z1 live service-channel ingress is implemented, published, and
|
|
smoke-passed on 2026-05-07. Node-agent/host-agent `0.2.182` artifacts,
|
|
Docker image `rap-node-agent:0.2.182`, release manifests, and update
|
|
policies are published. Backend `rap-backend:fabric-service-channel-0.2.182`
|
|
is deployed on docker-test. The runtime fix is a dynamic mesh listener
|
|
handler: synthetic config refreshes now update `/mesh/v1/forward`,
|
|
service-channel ingress, production routes, delivery inbox, and forward
|
|
transport without requiring a port/listener restart. Backend route-feedback
|
|
latest policy now prevents a fresh healthy heartbeat from immediately
|
|
overwriting active degraded/fenced feedback before TTL expiry, so rebuild
|
|
decisions survive long enough for nodes to apply them. Script
|
|
`scripts/fabric/c18z1-live-service-channel-ingress-smoke.ps1` posts signed
|
|
generic packet batches to the running `test-1` service-channel HTTP endpoint,
|
|
waits both entry and exit runtime configs, verifies exit inbox delivery,
|
|
injects route feedback, observes Control Plane rebuild, waits node
|
|
`applied_rebuild`, sends a second batch over the replacement route, and
|
|
expires both temporary route intents. Result:
|
|
`artifacts/c18z1-live-service-channel-ingress-smoke-result.json` run
|
|
`c18z1-20260507-203628`. All current nodes report `0.2.182/current` at the
|
|
last check.
|
|
- C18Z2 live service-channel sustained soak/failure smoke is implemented and
|
|
passed on 2026-05-07 without a new runtime release. Script
|
|
`scripts/fabric/c18z2-live-service-channel-soak-smoke.ps1` drives signed
|
|
generic packet batches through the running `test-1` service-channel HTTP
|
|
endpoint, keeps temporary primary/alternate `test-1 -> test-2` route intents
|
|
visible, restarts the exit-node container `rap_test_node_test_2`, waits for
|
|
the exit runtime to reload synthetic config, and verifies recovery batches
|
|
reach the exit fabric inbox after the restart. Result:
|
|
`artifacts/c18z2-live-service-channel-soak-smoke-result.json` run
|
|
`c18z2-20260507-205112`: warm batches `6/6`, during-restart batches `3/3`,
|
|
recovery batches `8/8`, exit inbox depth grew from post-restart baseline
|
|
`0` to `88`, drops `0`, and both temporary route intents expired.
|
|
- C18Z3 live service-channel entry/WebSocket/degraded-fallback smoke is
|
|
implemented, published, and passed on 2026-05-07. Node-agent/host-agent
|
|
`0.2.183` artifacts and Docker image `rap-node-agent:0.2.183` are published
|
|
to docker-test downloads; update policies for `test-1/2/3` are set to
|
|
`rolling` target `0.2.183`, and the test containers run that image. The
|
|
runtime fix makes the entry node honor the signed service-channel lease
|
|
authority: leases with `status=degraded_fallback` or
|
|
`primary_route.status=missing_route_intent` now force backend fallback instead
|
|
of reusing stale generic route candidates. The same fallback rule is applied
|
|
to HTTP and WebSocket packet ingress. Script
|
|
`scripts/fabric/c18z3-live-service-channel-entry-ws-fallback-smoke.ps1`
|
|
verifies signed HTTP warm batches, WebSocket ingress parity, entry-node
|
|
container restart while the lease exists, recovery batches over the same
|
|
lease, explicit degraded fallback for a no-route exit, and route-intent
|
|
expiry. Result:
|
|
`artifacts/c18z3-live-service-channel-entry-ws-fallback-smoke-result.json`
|
|
run `c18z3-20260507-211402`: warm `4/4`, WebSocket packets `8`, recovery
|
|
`4/4`, backend fallback queue `0 -> 8`, route failures `0`, and all checks
|
|
passed. During publication the first `0.2.183` Docker tar had a malformed
|
|
entrypoint and stale size/hash metadata; it was rebuilt, the latest tar alias
|
|
was replaced, and the release artifact row was corrected to sha256
|
|
`231286cf5860b22cf8ca6550f67f61b0ca4b5011ab9b09995bcabbafe883fee1`, size
|
|
`7261696`.
|
|
- C18Z4 live service-channel long-session pressure smoke is implemented and
|
|
passed on 2026-05-07 without a new runtime release beyond `0.2.183`. Script
|
|
`scripts/fabric/c18z4-live-service-channel-session-pressure-smoke.ps1` opens
|
|
one signed long-lived service-channel WebSocket from `test-1` to `test-2`,
|
|
sends 48 packet batches / 384 packets, expires the primary route intent while
|
|
the WebSocket session is still active, waits for dynamic synthetic-config
|
|
refresh, and verifies the remaining packets use the alternate route. Result:
|
|
`artifacts/c18z4-live-service-channel-session-pressure-smoke-result.json`
|
|
run `c18z4-20260507-212748`: exit inbox depth `0 -> 384`, route failure delta
|
|
`0`, flow drop delta `0`, backend fallback queue `0 -> 0`, primary route
|
|
removed from entry/exit configs, alternate route selected after the switch,
|
|
and both route intents expired. This proves the shared Fabric Service Channel
|
|
can keep a service session alive while Control Plane changes the live route
|
|
set, without falling back to backend relay.
|
|
- C18Z5 live service-channel exit-restart smoke is implemented and passed on
|
|
2026-05-07 without a new runtime release beyond `0.2.183`. Script
|
|
`scripts/fabric/c18z5-live-service-channel-exit-restart-smoke.ps1` keeps one
|
|
signed WebSocket service-channel session open from `test-1` to `test-2`,
|
|
sends pre-outage traffic, stops `test-2` for a bounded outage while traffic
|
|
continues, starts it again, waits runtime readiness, then sends recovery
|
|
traffic over the same WebSocket. Result:
|
|
`artifacts/c18z5-live-service-channel-exit-restart-smoke-result.json` run
|
|
`c18z5-20260507-213745`: pre/outage/recovery batches `12/24/24`, total
|
|
packets `480`, route failure delta `48`, backend fallback queue `0 -> 192`,
|
|
flow drop delta `0`, and recovery exit inbox `0 -> 192`. This proves real
|
|
exit-node failure is visible as fallback/failure telemetry while the
|
|
long-lived service channel remains usable and fabric delivery resumes after
|
|
the exit runtime returns. After the test, `test-2` and all active cluster
|
|
nodes were healthy/current on `0.2.183`.
|
|
- C18Z6 live service-channel active rebuild smoke is implemented and passed on
|
|
2026-05-07 without a new runtime release beyond `0.2.183`. Script
|
|
`scripts/fabric/c18z6-live-service-channel-active-rebuild-smoke.ps1` keeps a
|
|
signed WebSocket service-channel session open from `test-1` to `test-2`,
|
|
sends pre-rebuild traffic, injects route-health feedback that marks the
|
|
primary route stale and names the alternate route as replacement, waits for
|
|
Control Plane `rebuild_status=applied`, waits for node-agent
|
|
`route_manager_transition.status=applied_rebuild`, then continues sending
|
|
over the same WebSocket. Result:
|
|
`artifacts/c18z6-live-service-channel-active-rebuild-smoke-result.json` run
|
|
`c18z6-20260507-214900`: pre/post batches `16/32`, total packets `384`,
|
|
exit inbox depth `0 -> 384`, Control Plane replacement route
|
|
`b2f3c510-46d2-4dce-8389-3952a99d0311`, route failure delta `0`, flow drop
|
|
delta `0`, backend fallback queue `0 -> 0`, all checks passed, and all
|
|
active nodes remained healthy/current on `0.2.183`. This proves a live
|
|
service channel can apply a route-manager rebuild decision without rebuilding
|
|
the service WebSocket.
|
|
- C18Z7 live service-channel concurrent isolation smoke is implemented and
|
|
passed on 2026-05-07 without a new runtime release beyond `0.2.183`. Script
|
|
`scripts/fabric/c18z7-live-service-channel-concurrent-isolation-smoke.ps1`
|
|
opens three signed WebSocket service-channel sessions over the same
|
|
`test-1 -> test-2` entry/exit pair, interleaves packet batches across all
|
|
sessions, injects primary-route stale feedback, waits for Control Plane
|
|
`rebuild_status=applied` and node-agent `applied_rebuild`, then continues all
|
|
sessions over the same sockets. Result:
|
|
`artifacts/c18z7-live-service-channel-concurrent-isolation-smoke-result.json`
|
|
run `c18z7-20260507-215727`: 3 sessions, 36 rounds, 288 packets per session,
|
|
864 packets total, each session exit inbox depth `288`, total exit depth
|
|
`864`, backend fallback delta `0`, route failure delta `0`, flow drop delta
|
|
`0`, and all active nodes healthy/current on `0.2.183`. This proves rebuild
|
|
and route-manager state are shared correctly without one active service
|
|
session starving or poisoning the other concurrent sessions.
|
|
- C18Z8 live service-channel backpressure isolation smoke is implemented and
|
|
passed on 2026-05-07 without a new runtime release beyond `0.2.183`. Script
|
|
`scripts/fabric/c18z8-live-service-channel-backpressure-isolation-smoke.ps1`
|
|
opens two interactive signed WebSocket sessions plus one abusive session over
|
|
the same `test-1 -> test-2` entry/exit pair. The abusive session sends 1300
|
|
packets on one stable 5-tuple to force a single flow shard to hit bounded
|
|
queue pressure while the interactive sessions continue sending small batches.
|
|
Result:
|
|
`artifacts/c18z8-live-service-channel-backpressure-isolation-smoke-result.json`
|
|
run `c18z8-20260507-221347`: both interactive sessions delivered 192 packets
|
|
each, the abusive flow reached scheduler high watermark `1024`, scheduled
|
|
`1030` packets on the hottest channel, dropped `282` packets on that channel,
|
|
produced backend fallback delta `0`, route failure delta `0`, and all active
|
|
nodes stayed healthy/current on `0.2.183`. This proves bounded backpressure is
|
|
visible and isolated to the overloaded logical flow without starving other
|
|
active service sessions.
|
|
- C18Z9 route-pool runtime selection is implemented, released as node/host
|
|
agent `0.2.184`, published to docker-test downloads, and passed on
|
|
2026-05-07. Runtime fix: when Control Plane marks a service-channel route
|
|
`rebuild_status=applied` and provides `replacement_route_id`, node-agent now
|
|
treats that replacement as the preferred route for sticky flow/channel
|
|
selection instead of merely withdrawing the bad route and falling back to
|
|
config order. Unit coverage:
|
|
`TestFabricClientPacketIngressPrefersControlPlaneReplacementOverConfigOrder`.
|
|
Live script
|
|
`scripts/fabric/c18z9-live-service-channel-route-pool-smoke.ps1` creates a
|
|
route pool with slow relay primary `test-1 -> test-3 -> test-2` and fast
|
|
direct replacement `test-1 -> test-2`, keeps one signed WebSocket active,
|
|
injects stale-route feedback, waits for Control Plane and node-agent
|
|
`applied_rebuild`, then verifies the same service session continues over the
|
|
direct replacement. Result:
|
|
`artifacts/c18z9-live-service-channel-route-pool-smoke-result.json` run
|
|
`c18z9-20260507-224901`: 54 batches / 432 packets sent and delivered to exit,
|
|
backend fallback delta `0`, route failure delta `0`, flow drop delta `0`, and
|
|
temporary route intents expired. Test containers `test-1/2/3` run
|
|
`rap-node-agent:0.2.184`; `usa-los-1`, `home-1`, and
|
|
`ifcm-rufms-s-mo1cr` remain healthy on `0.2.183` until their rollout policy is
|
|
advanced.
|
|
- C18Z10 service-channel exit-pool failover is implemented, released as
|
|
node/host-agent `0.2.185`, published to docker-test downloads, registered in
|
|
the stable update channel, and passed on 2026-05-07. Backend service-channel
|
|
leases now bind signed entry/exit pools, selected exit follows the selected
|
|
primary route, and Control Plane replacement can cross to another authorized
|
|
exit when route intents share an exit-pool/resource metadata key. Node-agent
|
|
now honors the signed lease primary route as the initial service-channel
|
|
preference before normal config-order selection. Unit coverage:
|
|
`TestIssueFabricServiceChannelLeaseSelectsHealthyAlternateExitFromPool`,
|
|
`TestGetNodeSyntheticMeshConfigReplacesFencedServiceChannelRouteAcrossExitPool`,
|
|
and `TestFabricClientPacketIngressUsesLeasePreferredRouteBeforeConfigOrder`.
|
|
Live script
|
|
`scripts/fabric/c18z10-live-service-channel-exit-pool-smoke.ps1` creates a
|
|
primary exit route `test-1 -> test-2` and an alternate exit route
|
|
`test-1 -> test-3` in the same exit pool, keeps one signed WebSocket active,
|
|
verifies pre-rebuild traffic reaches the primary exit, injects stale-route
|
|
feedback, waits for Control Plane/node-agent `applied_rebuild`, then verifies
|
|
post-rebuild traffic reaches the alternate exit. Result:
|
|
`artifacts/c18z10-live-service-channel-exit-pool-smoke-result.json` run
|
|
`c18z10-20260507-232645`: 54 batches / 432 packets sent, primary exit queue
|
|
`144`, alternate exit queue `288`, backend fallback `0`, route failure delta
|
|
`0`, flow drop delta `0`, decision source
|
|
`service_channel_feedback_exit_pool_replacement`, and temporary route intents
|
|
expired. Backend and `test-1/2/3` are running `0.2.185`; update plans now
|
|
return download URLs on `192.168.200.61:18080` when the API is reached
|
|
directly on `18121`.
|
|
- C18Z11 service-channel entry-pool failover contract is implemented and
|
|
backend-deployed as `rap-backend:fabric-service-channel-0.2.186`; node-agent
|
|
remains `0.2.185` because no node runtime binary change was required.
|
|
Backend lease selection now keeps `selected_entry_node_id` aligned with the
|
|
selected primary route when the healthy route starts at another authorized
|
|
entry node. Route replacement scope also understands entry-pool metadata
|
|
keys (`entry_pool_id`, `service_entry_pool_id`, `fabric_entry_pool_id`) in
|
|
addition to exit-pool/resource keys, and route decision reports count
|
|
entry-pool replacement decisions. Unit coverage:
|
|
`TestIssueFabricServiceChannelLeaseSelectsHealthyAlternateEntryFromPool` and
|
|
`TestGetNodeSyntheticMeshConfigReplacesFencedServiceChannelRouteAcrossEntryPool`.
|
|
Live script
|
|
`scripts/fabric/c18z11-live-service-channel-entry-pool-smoke.ps1` creates
|
|
primary entry route `test-1 -> test-2` and alternate entry route
|
|
`test-3 -> test-2`, verifies the initial lease uses `test-1`, sends 144
|
|
packets, injects service-channel feedback fencing the primary entry route,
|
|
verifies a refreshed lease selects `test-3`, then sends 288 more packets
|
|
through the alternate entry to the same exit. Result:
|
|
`artifacts/c18z11-live-service-channel-entry-pool-smoke-result.json` run
|
|
`c18z11-20260507-235341`: exit queue `432`, backend fallback `0`, route
|
|
failure deltas `0/0`, flow drop deltas `0/0`, and temporary route intents
|
|
expired. This is a lease refresh/reconnect contract for entry replacement;
|
|
preserving a broken client-to-entry socket across an entry node outage is not
|
|
expected.
|
|
- C18Z12 service-channel route quality scoring is implemented and
|
|
backend-deployed as `rap-backend:fabric-service-channel-0.2.187`; node-agent
|
|
remains `0.2.185`. Backend now uses service-neutral runtime quality feedback
|
|
from `fabric_service_channel_runtime_report.ingress.flow_scheduler` when
|
|
scoring lease routes: `last_send_duration_ms` adds deterministic latency
|
|
boosts/penalties, and recent failures/stalls apply bounded penalties. This is
|
|
protocol-agnostic and applies to the shared fabric channel, not HTTP/RDP/DNS
|
|
special cases. Unit coverage:
|
|
`TestIssueFabricServiceChannelLeasePrefersFastHealthyRouteFeedback`. Live
|
|
script `scripts/fabric/c18z12-service-channel-route-quality-smoke.ps1`
|
|
creates a high-priority slow relay route `test-1 -> test-3 -> test-2` and a
|
|
lower-priority fast direct route `test-1 -> test-2`; the initial lease
|
|
selects the slow route by policy priority, then quality telemetry reports
|
|
fast route `8ms` and slow route `900ms`, and the refreshed lease selects the
|
|
fast route with score reason `service_channel_quality_latency_le_10ms`.
|
|
Result: `artifacts/c18z12-service-channel-route-quality-smoke-result.json`
|
|
run `c18z12-20260508-000209`; all checks passed and temporary route intents
|
|
expired.
|
|
- C18Z13 live service-channel route quality self-learning is implemented,
|
|
released as node-agent `0.2.188`, published to docker-test downloads,
|
|
registered in the stable update channel, and deployed to docker-test
|
|
containers `test-1/2/3`. Runtime fix: positive sub-millisecond
|
|
service-channel send durations are rounded to `1ms`, preventing fast local
|
|
routes from looking like "no quality sample". Unit coverage:
|
|
`TestFabricFlowSchedulerRoundsSubMillisecondSendDuration`. Live script
|
|
`scripts/fabric/c18z13-live-service-channel-route-quality-smoke.ps1` proves
|
|
the self-learning path without heartbeat injection: initial lease picks a
|
|
higher-priority relay route, real service-channel traffic sends 24 batches /
|
|
192 packets over the fast direct route, backend persists healthy route
|
|
feedback from the node-agent heartbeat (`last_send_duration_ms=1`,
|
|
`score_adjustment=90`), and a refreshed lease prefers that fast route over a
|
|
newly introduced higher-priority relay candidate. Result:
|
|
`artifacts/c18z13-live-service-channel-route-quality-smoke-result.json` run
|
|
`c18z13-20260508-001610`; backend fallback `0`, flow drops `0`, temporary
|
|
route intents expired. Published release id:
|
|
`64effc62-18b6-4eeb-a1c9-f5fb8e251491`.
|
|
- C18Z14 active-session route-quality preference is implemented. Backend
|
|
`rap-backend:fabric-service-channel-0.2.190` and node-agent `0.2.189` are
|
|
deployed to docker-test `test-1/2/3`; node-agent `0.2.189` is published to
|
|
docker-test downloads and registered in the stable update channel as release
|
|
`9bda9bac-71f3-4e8f-ae70-2abccb1cb866`. Backend now decays older healthy
|
|
service-channel feedback before lease scoring so stale success loses weight
|
|
before expiry. Node-agent consumes healthy route-quality observations from
|
|
signed synthetic config and can override sticky per-flow/config-order route
|
|
choice when a learned route is significantly better. Unit coverage:
|
|
`TestFabricClientPacketIngressQualityPreferenceOverridesStickyRoute` and
|
|
`TestIssueFabricServiceChannelLeaseDecaysOlderHealthyRouteFeedback`. Live
|
|
script
|
|
`scripts/fabric/c18z14-live-service-channel-active-quality-shift-smoke.ps1`
|
|
keeps one signed WebSocket open while route policy changes: it starts on a
|
|
higher-priority relay route, expires that route, sends real traffic through
|
|
the fast direct route to teach feedback, introduces a new higher-priority
|
|
relay candidate, and verifies the same active session stays on the learned
|
|
fast route. Result:
|
|
`artifacts/c18z14-live-service-channel-active-quality-shift-smoke-result.json`
|
|
run `c18z14-20260508-071644`; 60 batches / 480 packets delivered, backend
|
|
fallback `0`, flow drops `0`, temporary route intents expired.
|
|
- C18Z15 effective route-quality score telemetry is implemented. Backend
|
|
`rap-backend:fabric-service-channel-0.2.191` is deployed on docker-test, and
|
|
node-agent `0.2.190` is built, published to docker-test downloads, registered
|
|
in the stable update channel, and deployed to `test-1/2/3`. Published release
|
|
id: `2e4cd0c8-2480-4637-b845-6dcb115dbebd`. Backend feedback reports now
|
|
include decayed `effective_score_adjustment` alongside raw
|
|
`score_adjustment`; node-agent consumes the effective score for active
|
|
route-quality preference and exposes sorted `route_quality_preferences` in
|
|
runtime telemetry with raw/effective score and decay reasons. Unit coverage:
|
|
`TestFabricClientPacketIngressQualityPreferenceUsesEffectiveScore` and
|
|
`TestServiceChannelRouteFeedbackReportIncludesEffectiveDecayedScore`. Live
|
|
script
|
|
`scripts/fabric/c18z15-live-service-channel-effective-quality-smoke.ps1`
|
|
verifies route-quality preference telemetry, effective score visibility, and
|
|
decayed effective score visibility after the active-session quality-shift
|
|
scenario. Result:
|
|
`artifacts/c18z15-live-service-channel-effective-quality-smoke-result.json`
|
|
run `c18z14-20260508-073538`; 60 batches / 480 packets delivered, backend
|
|
fallback `0`, flow drops `0`, temporary route intents expired.
|
|
- C18Z16 per-channel route-quality fairness telemetry is implemented. Node-agent
|
|
`0.2.191` is built, published to docker-test downloads, registered in the
|
|
stable update channel, and deployed to `test-1/2/3`; backend remains
|
|
`rap-backend:fabric-service-channel-0.2.191`. Published release id:
|
|
`f072759c-5c3b-4ba0-936a-f59b6d3d7632`. Flow-scheduler channel stats now
|
|
expose the applied `quality_preference_route_id`, effective/raw preference
|
|
score, and preference reasons, so operators can see which logical channels
|
|
actually used learned route quality. Unit coverage:
|
|
`TestFabricClientPacketIngressQualityPreferencePreservesMultiChannelFairness`.
|
|
Live script
|
|
`scripts/fabric/c18z16-live-service-channel-quality-fairness-smoke.ps1`
|
|
validates multi-channel quality-preference fairness after the active-session
|
|
route-quality shift. Result:
|
|
`artifacts/c18z16-live-service-channel-quality-fairness-smoke-result.json`
|
|
run `c18z14-20260508-074943`; 60 batches / 480 packets delivered, 32 served
|
|
logical channels, 32 channels with quality preference applied, backend
|
|
fallback `0`, flow drops `0`, temporary route intents expired.
|
|
- C18Z17 stale route-quality marker cleanup is implemented. Node-agent
|
|
`0.2.192` is built, published to docker-test downloads, registered in the
|
|
stable update channel, and deployed to `test-1/2/3`; backend remains
|
|
`rap-backend:fabric-service-channel-0.2.191`. Published release id:
|
|
`846881bd-e7e0-4212-b8c9-4a6012c6eff7`. Flow-scheduler channel stats now
|
|
clear quality preference markers when the preference is no longer in the
|
|
effective preference set or when the route manager withdraws that route. Unit
|
|
coverage:
|
|
`TestFabricClientPacketIngressClearsStaleQualityPreferenceMarkers` and
|
|
`TestFabricClientPacketIngressClearsWithdrawnQualityPreferenceMarkers`.
|
|
Live script
|
|
`scripts/fabric/c18z17-live-service-channel-quality-cleanup-smoke.ps1`
|
|
verifies cleanup after the active-session quality/fairness scenario. Result:
|
|
`artifacts/c18z17-live-service-channel-quality-cleanup-smoke-result.json`
|
|
run `c18z14-20260508-075750`; 60 batches / 480 packets delivered, active
|
|
quality markers `32`, stale quality markers `0`, visible preferences `3`,
|
|
backend fallback `0`, flow drops `0`, temporary route intents expired.
|
|
- C18Z18 service-session-scoped flow scheduler memory is implemented.
|
|
Node-agent `0.2.193` is built, published to docker-test downloads,
|
|
registered in the stable update channel, and deployed to `test-1/2/3`;
|
|
backend remains `rap-backend:fabric-service-channel-0.2.191`. Published
|
|
release id: `05a3d29e-8a62-4bc8-84a3-1d00b794b9c9`. Runtime-sent flow
|
|
scheduler channel keys now include the VPN/service session:
|
|
`vpn:{vpnConnectionID}:flow-NN`. This keeps route memory, failed-route
|
|
avoidance, served/drop counters, and route-quality markers isolated when
|
|
several service-channel sessions share one entry/exit and hash to the same
|
|
logical flow shard. Unit coverage:
|
|
`TestFabricClientPacketIngressIsolatesRouteMemoryPerVPNConnection` and
|
|
`TestFabricClientPacketIngressQualityPreferencePreservesMultiChannelFairness`.
|
|
Live script
|
|
`scripts/fabric/c18z18-service-channel-session-scoped-fairness-smoke.ps1`
|
|
wraps the live C18Z17 quality path and verifies served live channels are
|
|
session-scoped, unscoped served `flow-NN` channels are absent, quality
|
|
markers are session-scoped, backend fallback is `0`, and flow drops are `0`.
|
|
Result:
|
|
`artifacts/c18z18-service-channel-session-scoped-fairness-smoke-result.json`
|
|
run `c18z14-20260508-082520`; 60 batches / 480 packets delivered, served
|
|
channels `32`, session-scoped served channels `32`, session-scoped quality
|
|
channels `32`, unscoped served channels `0`, backend fallback `0`, flow drops
|
|
`0`, temporary route intents expired.
|
|
- C18Z19 bounded parallel logical-flow send window is implemented. Node-agent
|
|
`0.2.194` is built, published to docker-test downloads, registered in the
|
|
stable update channel, and deployed to `test-1/2/3`; backend remains
|
|
`rap-backend:fabric-service-channel-0.2.191`. Published release id:
|
|
`926e5b84-4b0b-4f47-b1fe-798d8105679f`. The live node-agent runtime enables
|
|
`MaxParallelFlowSends=4`, so independent scheduled logical channels can send
|
|
concurrently instead of one slow channel blocking all following channels.
|
|
This remains service-neutral and does not inspect HTTP/RDP/DNS/application
|
|
traffic. Telemetry now exposes `max_parallel_flow_sends` and
|
|
`send_flow_parallel_batches`. Unit coverage:
|
|
`TestFabricClientPacketIngressParallelFlowWindowDoesNotBlockIndependentChannel`.
|
|
Live script
|
|
`scripts/fabric/c18z19-service-channel-parallel-flow-window-smoke.ps1` wraps
|
|
the C18Z18 live route-quality/session-scoped path and verifies the parallel
|
|
window is enabled and observed while backend fallback and flow drops stay at
|
|
zero. Result:
|
|
`artifacts/c18z19-service-channel-parallel-flow-window-smoke-result.json`
|
|
run `c18z14-20260508-084133`; 60 batches / 480 packets delivered,
|
|
`max_parallel_flow_sends=4`, `send_flow_parallel_batches=60`, served
|
|
channels `32`, session-scoped quality channels `32`, backend fallback `0`,
|
|
flow drops `0`, temporary route intents expired.
|
|
- C18Z20 per-channel latency/retry/in-flight telemetry and adaptive recommended
|
|
send-window telemetry are implemented. Node-agent `0.2.195` is built,
|
|
published to docker-test downloads, registered in the stable update channel,
|
|
and deployed to `test-1/2/3`; backend remains
|
|
`rap-backend:fabric-service-channel-0.2.191`. Published release id:
|
|
`b9e198e0-e012-4600-ad14-856820aff41c`. Scheduler telemetry now includes
|
|
global `in_flight`, `max_in_flight`, slow/failing channel counts, and
|
|
per-channel `send_attempts`, `send_successes`, `send_failures`,
|
|
`in_flight`, `max_in_flight`, and latency buckets. Ingress telemetry now
|
|
includes `recommended_parallel_flow_sends`; the recommendation shrinks under
|
|
bounded drops, degraded fallback recommendations, repeated failures, or
|
|
slow/stalled channels. Unit coverage:
|
|
`TestFabricFlowSchedulerRecommendsSmallerWindowUnderPressure` and
|
|
`TestFabricClientPacketIngressParallelFlowWindowDoesNotBlockIndependentChannel`.
|
|
Live script
|
|
`scripts/fabric/c18z20-service-channel-adaptive-window-telemetry-smoke.ps1`
|
|
wraps the C18Z19 live path and verifies the new telemetry on real docker-test
|
|
nodes. Result:
|
|
`artifacts/c18z20-service-channel-adaptive-window-telemetry-smoke-result.json`
|
|
run `c18z14-20260508-085635`; 60 batches / 480 packets delivered,
|
|
`max_parallel_flow_sends=4`, `recommended_parallel_flow_sends=4`,
|
|
`scheduler_max_in_flight=4`, attempts/success/latency visible on 32 channels,
|
|
backend fallback `0`, flow drops `0`, temporary route intents expired.
|
|
- C18Z21 rolling per-channel/session quality windows are implemented.
|
|
Node-agent `0.2.196` is built, published to docker-test downloads,
|
|
registered in the stable update channel, and deployed to `test-1/2/3`;
|
|
backend remains `rap-backend:fabric-service-channel-0.2.191`. Published
|
|
release id: `813b2050-4d4e-444c-9bde-72b1d1f7dd35`. Scheduler decisions now
|
|
use a bounded fresh quality window instead of lifetime-only drop/failure
|
|
counters, so old pressure rolls out after newer successful samples. Telemetry
|
|
now exposes scheduler-level `quality_window_sample_count`,
|
|
`quality_window_failure_count`, `quality_window_slow_count`,
|
|
`quality_window_drop_count`, and per-channel success/failure/slow/drop sample
|
|
counts, average latency, and last update time. Unit coverage:
|
|
`TestFabricFlowSchedulerRollingQualityWindowForgetsOldPressure`,
|
|
`TestFabricFlowSchedulerRecommendsSmallerWindowUnderPressure`, and
|
|
`TestFabricClientPacketIngressParallelFlowWindowDoesNotBlockIndependentChannel`.
|
|
Live script
|
|
`scripts/fabric/c18z21-service-channel-rolling-quality-window-smoke.ps1`
|
|
wraps the C18Z20 live path and verifies the rolling-window telemetry on real
|
|
docker-test nodes. Result:
|
|
`artifacts/c18z21-service-channel-rolling-quality-window-smoke-result.json`
|
|
run `c18z14-20260508-091952`; 60 batches / 480 packets delivered,
|
|
scheduler quality-window samples `480`, failures `0`, drops `0`, window
|
|
samples/success/latency visible on 32 channels, `recommended_parallel_flow_sends=4`,
|
|
backend fallback `0`, flow drops `0`, temporary route intents expired.
|
|
- C18Z22 backend durable route feedback now consumes the rolling quality
|
|
window from node-agent heartbeat metadata. Backend
|
|
`rap-backend:fabric-service-channel-0.2.197` is built and deployed on
|
|
docker-test; node-agent remains `0.2.196` on `test-1/2/3`. For agents that
|
|
expose `quality_window_*`, backend uses fresh rolling failure/drop/slow
|
|
counts and rolling average latency when creating `fabric_service_channel`
|
|
route feedback; old `last_failed_route_id`, `consecutive_failures`, and
|
|
`stall_count` remain fallback inputs for older agents only. This prevents old
|
|
route failures from dominating durable scoring after the channel has recovered
|
|
with a clean rolling window. Unit coverage:
|
|
`TestRecordHeartbeatUsesRollingQualityWindowForRouteFeedback` and
|
|
`TestRecordHeartbeatPersistsServiceChannelRouteFeedbackForLaterLease`.
|
|
Live script
|
|
`scripts/fabric/c18z22-service-channel-rolling-feedback-smoke.ps1` wraps the
|
|
C18Z21 live path and verifies persisted route feedback contains
|
|
`service_channel_rolling_quality_window` plus payload `quality_window_*`
|
|
fields. Result:
|
|
`artifacts/c18z22-service-channel-rolling-feedback-smoke-result.json` run
|
|
`c18z14-20260508-093100`; 60 batches / 480 packets delivered, route feedback
|
|
count `1`, rolling feedback count `1`, healthy rolling feedback count `1`,
|
|
rolling payload count `1`, backend fallback `0`, flow drops `0`.
|
|
- C18Z23 recovery hysteresis is implemented for recovered service-channel
|
|
routes. Backend `rap-backend:fabric-service-channel-0.2.198` is built and
|
|
deployed on docker-test; node-agent remains `0.2.196` on `test-1/2/3`.
|
|
When a route has an operator-expire/manual retry cooldown from prior fenced
|
|
feedback but now also has healthy rolling-window feedback, backend re-admits
|
|
the route as `authorized` while applying a bounded recovery hysteresis score
|
|
penalty (`150`) and `service_channel_recovery_hysteresis` reason. This keeps
|
|
recovered routes available as alternates without immediately displacing a
|
|
steady route and reducing route-selection flapping. Unit coverage:
|
|
`TestIssueFabricServiceChannelLeaseDampensRecoveredRouteDuringRetryCooldown`
|
|
and `TestRecordHeartbeatUsesRollingQualityWindowForRouteFeedback`. Live
|
|
script
|
|
`scripts/fabric/c18z23-service-channel-recovery-hysteresis-smoke.ps1` wraps
|
|
the C18Z22 live path and verifies backend `0.2.198`, rolling feedback, and
|
|
clean live forwarding. Result:
|
|
`artifacts/c18z23-service-channel-recovery-hysteresis-smoke-result.json` run
|
|
`c18z14-20260508-094111`; 60 batches / 480 packets delivered, backend
|
|
fallback `0`, flow drops `0`, recovery hysteresis penalty `150`.
|
|
- C18Z24 recovery visibility is implemented for service-channel route
|
|
diagnostics. Backend `rap-backend:fabric-service-channel-0.2.199` is built
|
|
and deployed on docker-test; node-agent remains `0.2.196` on `test-1/2/3`.
|
|
Route feedback API responses and node-scoped service-channel feedback reports
|
|
now expose `recovery_state`, `recovery_hysteresis_active`, and
|
|
`recovery_hysteresis_penalty`, while route path decision reports count
|
|
`recovery_hysteresis_count`. Admin diagnostics now show recovered/hysteresis
|
|
chips and a recovery column beside route feedback status. Unit coverage:
|
|
`TestIssueFabricServiceChannelLeaseDampensRecoveredRouteDuringRetryCooldown`,
|
|
`TestServiceChannelRouteFeedbackReportExposesRecoveryState`, and
|
|
`TestRoutePathDecisionReportCountsRecoveryHysteresis`. Smoke result:
|
|
`artifacts/c18z24-service-channel-recovery-visibility-smoke-result.json`;
|
|
route feedback API exposed recovery shape for 109 observations, backend
|
|
image `0.2.199` was live, and the web-admin build was published to
|
|
`rap_web_admin`.
|
|
- C18Z25 recovery promotion policy is implemented. Backend
|
|
`rap-backend:fabric-service-channel-0.2.200` is built and deployed on
|
|
docker-test; node-agent remains `0.2.196`. A route under manual retry
|
|
cooldown remains `recovered` with hysteresis penalty until it reports at
|
|
least 64 clean rolling-window samples (`success >= 64`, failures/slow/drops
|
|
zero). After that it is promoted back to steady `healthy`, gets
|
|
`recovery_promoted=true`, `service_channel_recovery_promoted`, and no
|
|
hysteresis penalty. Admin/API now expose promoted counts/flags alongside
|
|
recovered/hysteresis state. Smoke result:
|
|
`artifacts/c18z25-service-channel-recovery-promotion-smoke-result.json`;
|
|
backend image `0.2.200` was live and route-feedback API exposed recovery
|
|
state for 109 observations.
|
|
- C18Z26 recovery demotion policy is implemented. Backend
|
|
`rap-backend:fabric-service-channel-0.2.201` is built and deployed on
|
|
docker-test; node-agent remains `0.2.196`. If a previously recovered or
|
|
promoted route under retry cooldown reports fresh rolling failures, drops,
|
|
slow samples, degraded fallback, rebuild recommendation, or fenced feedback,
|
|
backend now exposes `recovery_demoted=true` with a concrete
|
|
`recovery_reason` such as `service_channel_recovery_demoted_failure`,
|
|
`..._slow`, `..._rebuild`, or `..._fenced`. Route score reasons include
|
|
`service_channel_recovery_demoted` and the specific demotion reason, and
|
|
route path decision reports count `recovery_demoted_count`. Admin diagnostics
|
|
now show demoted feedback/path chips and the demotion reason. Smoke result:
|
|
`artifacts/c18z26-service-channel-recovery-demotion-smoke-result.json`;
|
|
backend image `0.2.201` was live and route-feedback API exposed recovery
|
|
state for 109 observations.
|
|
- C18Z27 recovery policy tuning is implemented. Backend
|
|
`rap-backend:fabric-service-channel-0.2.202` is built and deployed on
|
|
docker-test; node-agent remains `0.2.196`. Effective service-channel
|
|
recovery policy now has a strict default contract and optional cluster
|
|
metadata override at `fabric_service_channel_recovery_policy`. API endpoints
|
|
`GET/PUT /clusters/{clusterID}/fabric/service-channels/recovery-policy`
|
|
expose and update hysteresis penalty, promotion minimum samples, demotion
|
|
thresholds for failures/drops/slow samples, and rebuild/fenced demotion
|
|
toggles. Lease route selection, route feedback reports, and node-scoped
|
|
synthetic config feedback consume the effective policy. Web-admin shows and
|
|
edits the policy in the service-channel route feedback card. Smoke result:
|
|
`artifacts/c18z27-service-channel-recovery-policy-smoke-result.json`; live
|
|
API updated policy values, then restored strict defaults
|
|
(`penalty=150`, `promotion_min_samples=64`, demotion thresholds `1`).
|
|
- C18Z28 recovery policy provenance is implemented. Backend
|
|
`rap-backend:fabric-service-channel-0.2.203` is built and deployed on
|
|
docker-test; node-agent remains `0.2.196`. `FabricServiceChannelRoute`,
|
|
`FabricServiceChannelLease`, signed lease authority payloads,
|
|
service-channel route feedback reports, and route path decision reports now
|
|
carry the effective recovery policy used for scoring and recovery decisions.
|
|
This makes every primary/alternate/fallback choice auditable against the
|
|
policy source and thresholds that produced it. Web-admin node diagnostics
|
|
show the service-channel feedback policy and route decision policy source.
|
|
Smoke result:
|
|
`artifacts/c18z28-service-channel-recovery-policy-provenance-smoke-result.json`;
|
|
live synthetic config and live lease issuance both exposed recovery policy
|
|
provenance on docker-test.
|
|
- C18Z29 feedback provenance guardrails are implemented. Backend
|
|
`rap-backend:fabric-service-channel-0.2.204` is built and deployed on
|
|
docker-test; node-agent remains `0.2.196`. Recovery policy now has a stable
|
|
fingerprint. Backend recognizes optional runtime feedback provenance fields
|
|
(`recovery_policy_fingerprint`, `route_generation`, `route_policy_version`,
|
|
`policy_version`), exposes observed/effective fingerprints/generations on
|
|
route feedback observations, and reports missing/stale counters. Explicit
|
|
stale policy/generation feedback is scored conservatively, cannot fence a
|
|
current route, and cannot request rebuild/demotion; missing provenance stays
|
|
compatible for current old agents but is visible in diagnostics. Web-admin
|
|
shows provenance warnings in service-channel feedback. Smoke result:
|
|
`artifacts/c18z29-service-channel-feedback-provenance-guard-smoke-result.json`.
|
|
- C18Z30 node-agent feedback provenance is implemented. Backend
|
|
`rap-backend:fabric-service-channel-0.2.209` and node-agent `0.2.208` are
|
|
built and deployed on docker-test (`test-1/2/3`). Node-agent now preserves the
|
|
signed synthetic config contract for recovery feedback/route decision fields
|
|
and records per-flow `recovery_policy_fingerprint`, `route_policy_version`,
|
|
and `route_generation` at send time, so feedback remains auditable even after
|
|
route churn/expiry. Backend heartbeat parsing now preserves those fields into
|
|
durable service-channel feedback payloads. Live smoke passed with 28/28
|
|
runtime channel stats carrying provenance, 3/3 feedback observations carrying
|
|
provenance, and no missing/stale provenance counters. Artifacts:
|
|
`artifacts/c18z30-node-telemetry-provenance-live-smoke-base-result.json` and
|
|
`artifacts/c18z30-node-agent-feedback-provenance-smoke-result.json`.
|
|
- C18Z31 service-channel rebuild ledger is implemented. Backend
|
|
`rap-backend:fabric-service-channel-0.2.211` is built and deployed on
|
|
docker-test; node-agent remains `0.2.208` on `test-1/2/3`. Backend now keeps
|
|
durable route rebuild attempt history in
|
|
`fabric_service_channel_route_rebuild_attempts`, upserted from synthetic
|
|
config route decisions when service-channel feedback requests rebuild. The
|
|
ledger stores trigger/rebuild status, old route, selected replacement,
|
|
policy fingerprint, generation, feedback status/reasons, latency/failure
|
|
counters, outcome, and compact decision payload. API endpoint
|
|
`GET /clusters/{clusterID}/fabric/service-channels/rebuild-attempts` exposes
|
|
the history; web-admin loads it into Service-channel route feedback
|
|
diagnostics as a rebuild ledger table. Migration `000026` is applied on
|
|
docker-test. Live smoke passed:
|
|
`artifacts/c18z31-base-active-rebuild-smoke-result.json` and
|
|
`artifacts/c18z31-service-channel-rebuild-ledger-smoke-result.json`.
|
|
- C18Z32 service-channel rebuild timeline is implemented. Backend
|
|
`rap-backend:fabric-service-channel-0.2.213` is built and deployed on
|
|
docker-test; node-agent remains `0.2.208` on `test-1/2/3`. The rebuild
|
|
attempts API now enriches durable ledger rows with node-agent heartbeat
|
|
correlation: matching `route_manager_transition`, route-generation apply or
|
|
withdrawn decision, post-rebuild selected route, flow packet/drop/failure
|
|
counters, and a compact chronological `timeline` with
|
|
`backend_decision`, `node_route_generation_apply`,
|
|
`node_route_manager_transition`, and `post_rebuild_traffic` stages. Matching
|
|
is generation-strict when the backend attempt has a generation, preventing
|
|
stale transition/status matches. Web-admin rebuild ledger shows backend,
|
|
agent, route-generation, and traffic columns. Live smoke passed:
|
|
`artifacts/c18z32-base-rebuild-ledger-smoke-result.json` and
|
|
`artifacts/c18z32-service-channel-rebuild-timeline-smoke-result.json`.
|
|
- C18Z33 service-channel rebuild guardrails are implemented. Backend
|
|
`rap-backend:fabric-service-channel-0.2.214` is built and deployed on
|
|
docker-test; node-agent remains `0.2.208`. Rebuild attempts API now adds
|
|
computed guard fields: `guard_status`, `guard_severity`, `guard_reason`,
|
|
age, and transition/traffic deadlines. Successful correlated rebuilds report
|
|
`guard_status=ok`, `guard_severity=good`; missing node transition,
|
|
route-generation correlation, post-rebuild traffic, unexpected selected
|
|
route, or post-rebuild drops/failures surface as warn/bad states. Web-admin
|
|
shows guard chips and counts in the service-channel rebuild ledger. Live
|
|
smoke passed: `artifacts/c18z33-base-rebuild-ledger-smoke-result.json` and
|
|
`artifacts/c18z33-service-channel-rebuild-guard-smoke-result.json`.
|
|
- C18Z34 service-channel rebuild health summary is implemented. Backend
|
|
`rap-backend:fabric-service-channel-0.2.215` is built and deployed on
|
|
docker-test; node-agent remains `0.2.208`. New endpoint
|
|
`GET /clusters/{clusterID}/fabric/service-channels/rebuild-health` returns a
|
|
cluster-level operational summary over the durable rebuild ledger/timeline:
|
|
counts by guard status/severity, applied/pending counts, affected reporter
|
|
nodes/routes, most recent bad attempts, and recommended operator action.
|
|
Web-admin shows the summary as a Rebuild health subpanel above the rebuild
|
|
ledger. Live smoke passed:
|
|
`artifacts/c18z34-base-rebuild-guard-smoke-result.json` and
|
|
`artifacts/c18z34-service-channel-rebuild-health-smoke-result.json`.
|
|
- C18Z35 service-channel rebuild alert silence lifecycle is implemented.
|
|
Backend `rap-backend:fabric-service-channel-0.2.216` is built and deployed on
|
|
docker-test; node-agent remains `0.2.208`. Migration `000027` creates
|
|
`fabric_service_channel_rebuild_alert_silences`, applied on docker-test. New
|
|
API `POST /clusters/{clusterID}/fabric/service-channels/rebuild-health/silences`
|
|
records bounded operator silence for an exact alert fingerprint:
|
|
reporter node, route, guard status, and generation. Rebuild health now
|
|
separates total bad/warn from active bad/warn and silenced counts; silenced
|
|
alerts are omitted from affected nodes/routes and active bad attempt lists.
|
|
A new generation, route, or reporter remains active by design. Web-admin
|
|
exposes `silence 6h` on active bad rebuild-health rows. Live smoke passed:
|
|
`artifacts/c18z35-base-rebuild-health-smoke-result.json` and
|
|
`artifacts/c18z35-service-channel-rebuild-alert-silence-smoke-result.json`.
|
|
- C18Z36 service-channel rebuild alert resurfacing is implemented. Backend
|
|
`rap-backend:fabric-service-channel-0.2.217` is built and deployed on
|
|
docker-test; node-agent remains `0.2.208`. Rebuild health marks active
|
|
bad/warn attempts as `alert_resurfaced` when an active silence exists for the
|
|
same reporter node, route, and guard status but a different generation. The
|
|
summary exposes `resurfaced_count` and `resurfaced_attempts`, including the
|
|
previous silenced generation and silence expiry. Web-admin shows a resurfaced
|
|
chip/table and allows silencing the new generation separately. Live smoke
|
|
passed: `artifacts/c18z36-base-rebuild-health-smoke-result.json` and
|
|
`artifacts/c18z36-service-channel-rebuild-alert-resurface-smoke-result.json`.
|
|
- C18Z37 service-channel readiness gate is implemented. Backend
|
|
`rap-backend:fabric-service-channel-0.2.218` is built and deployed on
|
|
docker-test; node-agent remains `0.2.208`. New endpoint
|
|
`GET /clusters/{clusterID}/fabric/service-channels/readiness` returns a fast
|
|
recent-window verdict: `clean`, `degraded`, or `blocked`, with active
|
|
bad/warn counts, resurfaced/silenced counts, missing transition,
|
|
route-generation, post-rebuild traffic, unexpected-route, and post-rebuild
|
|
degraded counters plus blocking/degraded reasons and recommended operator
|
|
action. Web-admin shows this as a top-level readiness panel in
|
|
Service-channel route feedback. Readiness and default admin health queries
|
|
are intentionally capped to a small recent window so the operator view stays
|
|
responsive after many rebuild attempts; deep ledger diagnostics remain a
|
|
separate next layer. Live smoke passed:
|
|
`artifacts/c18z37-base-rebuild-health-smoke-result.json` and
|
|
`artifacts/c18z37-service-channel-readiness-smoke-result.json`.
|
|
- C18Z38 service-channel rebuild ledger enrichment split is implemented.
|
|
Backend `rap-backend:fabric-service-channel-0.2.219` is built and deployed
|
|
on docker-test; node-agent remains `0.2.208`. The rebuild attempts API now
|
|
defaults to `enrichment=summary`, returning durable ledger rows without the
|
|
expensive heartbeat/timeline guard correlation. Operators can request
|
|
`enrichment=deep` explicitly for per-route investigation. Web-admin defaults
|
|
to the fast ledger, shows timeline/guard fields as deep-only in summary mode,
|
|
and provides a manual deep ledger toggle. C18Z32/C18Z33 smokes now request
|
|
deep enrichment. Live smoke passed:
|
|
`artifacts/c18z38-service-channel-rebuild-ledger-enrichment-smoke-result.json`.
|
|
- C18Z39 service-channel rebuild ledger drilldown is implemented. Backend
|
|
`rap-backend:fabric-service-channel-0.2.220` is built and deployed on
|
|
docker-test; node-agent remains `0.2.208`. The rebuild attempts API now
|
|
accepts `generation` and `offset`, allowing narrow deep investigations by
|
|
reporter node, route, service class, and route generation with bounded
|
|
pagination. Web-admin adds rebuild ledger filters for reporter/route/
|
|
generation/service plus prev/next paging in deep mode. Live smoke passed:
|
|
`artifacts/c18z39-service-channel-rebuild-ledger-drilldown-smoke-result.json`.
|
|
- C18Z40 service-channel rebuild incident grouping is implemented. Backend
|
|
`rap-backend:fabric-service-channel-0.2.222` is built and deployed on
|
|
docker-test; node-agent remains `0.2.208`. New endpoint
|
|
`GET /clusters/{clusterID}/fabric/service-channels/rebuild-incidents`
|
|
groups the bounded recent rebuild window by reporter node, route, service
|
|
class, generation, and guard status, exposing first/last seen, attempt count,
|
|
latest guard/replacement/outcome, silence/resurface flags, and recommended
|
|
action. The incident window is capped to 5 to keep default admin refresh
|
|
bounded; broader investigation still uses filtered deep ledger. Web-admin
|
|
shows a Rebuild incidents list and `open deep` loads the exact filtered deep
|
|
ledger slice for that incident. Live smoke passed:
|
|
`artifacts/c18z40-service-channel-rebuild-incidents-smoke-result.json`.
|
|
- C18Z41 service-channel rebuild incident actions are implemented. Backend
|
|
`rap-backend:fabric-service-channel-0.2.223` is built and deployed on
|
|
docker-test; node-agent remains `0.2.208`. New API
|
|
`POST /clusters/{clusterID}/fabric/service-channels/rebuild-incidents/investigations`
|
|
records an audit event when an operator opens a deep rebuild investigation.
|
|
Web-admin incident rows now expose `open deep` with audit and `silence 6h`
|
|
using the incident fingerprint fields; after silence the panel refreshes only
|
|
rebuild health/readiness/incidents instead of the whole cluster scope. Live
|
|
smoke passed:
|
|
`artifacts/c18z41-service-channel-rebuild-incident-actions-smoke-result.json`.
|
|
- C18Z42 service-channel rebuild correlation snapshots are implemented.
|
|
Backend `rap-backend:fabric-service-channel-0.2.224` is built and deployed
|
|
on docker-test; node-agent remains `0.2.208`. Migration `000028` adds
|
|
durable correlation/guard snapshot columns to
|
|
`fabric_service_channel_route_rebuild_attempts`, including node transition,
|
|
route-generation, post-rebuild traffic, guard status/severity/reason,
|
|
compact timeline, and `correlation_snapshot_at`. Deep enrichment now writes
|
|
the snapshot once; later deep/readiness/health/incidents reuse it and only
|
|
recompute age-sensitive guard state without scanning heartbeat history.
|
|
External summary ledger still strips guard/timeline fields to preserve the
|
|
fast C18Z38 contract. On docker-test, applying `000028` manually was required
|
|
before smoke because this manual backend redeploy path does not auto-apply
|
|
migrations. Live smoke passed twice; after warm snapshot timings were roughly
|
|
summary 92 ms, deep 2 ms, incidents 2 ms:
|
|
`artifacts/c18z42-service-channel-rebuild-correlation-snapshot-smoke-result.json`.
|
|
- C18Z43 service-channel schema preflight is implemented. Backend
|
|
`rap-backend:fabric-service-channel-0.2.225` is built and deployed on
|
|
docker-test; web-admin is redeployed. New endpoint
|
|
`GET /clusters/{clusterID}/fabric/service-channels/schema-status` checks the
|
|
DB relation/columns required by migration `000028` before operators rely on
|
|
rebuild health/readiness/incidents. Web-admin shows a Fabric schema preflight
|
|
panel beside service-channel readiness, with required/missing check counts and
|
|
operator action. Live smoke passed:
|
|
`artifacts/c18z43-service-channel-schema-preflight-smoke-result.json`.
|
|
- C18Z44 service-channel rebuild snapshot warmup is implemented. Backend
|
|
`rap-backend:fabric-service-channel-0.2.226` is built and deployed on
|
|
docker-test; web-admin is redeployed. New endpoint
|
|
`POST /clusters/{clusterID}/fabric/service-channels/rebuild-snapshots/warmup`
|
|
performs a bounded proactive pass over recent rebuild attempts. It fills
|
|
missing correlation snapshots, counts stale snapshots, and defers heavy stale
|
|
rescans because age-sensitive guard state is already recomputed from cached
|
|
snapshots on read. Web-admin adds a `warm snapshots` action and displays
|
|
warmed/fresh/missing/stale/deferred/error counts. Live smoke passed:
|
|
`artifacts/c18z44-service-channel-rebuild-snapshot-warmup-smoke-result.json`.
|
|
- C18Z45 service-channel rebuild snapshot auto-warmup is implemented. Backend
|
|
`rap-backend:fabric-service-channel-0.2.227` is built and deployed on
|
|
docker-test; node-agent remains `0.2.208`. Heartbeat processing now performs a
|
|
bounded missing-snapshot maintenance pass for the reporting node's recent
|
|
rebuild attempts. It only persists a snapshot when the heartbeat contains
|
|
runtime evidence such as post-rebuild traffic or matched route-manager/
|
|
route-generation state, preventing backend-only timelines from becoming stale
|
|
cache entries. Auto-warmup writes an audit event
|
|
`fabric.service_channel_rebuild_snapshot.auto_warmup` with trigger, heartbeat,
|
|
warmed route IDs, generations, rebuild IDs, counts, and errors. Live smoke
|
|
passed:
|
|
`artifacts/c18z45-service-channel-rebuild-snapshot-auto-warmup-smoke-result.json`.
|
|
- C18Z46 service-channel rebuild snapshot maintenance health is implemented.
|
|
Backend `rap-backend:fabric-service-channel-0.2.228` is built and deployed
|
|
on docker-test; web-admin is redeployed. New endpoint
|
|
`GET /clusters/{clusterID}/fabric/service-channels/rebuild-snapshots/health`
|
|
exposes bounded snapshot-cache maintenance status: recent attempt count,
|
|
valid/missing/overdue runtime-evidence snapshots, heartbeat threshold, latest
|
|
auto-warmup audit summary, and per-node warmed/error/missing counts. Web-admin
|
|
adds a `Snapshot maintenance` panel beside schema/readiness. Live smoke
|
|
passed:
|
|
`artifacts/c18z46-service-channel-rebuild-snapshot-health-smoke-result.json`.
|
|
- C18Z47 service-channel signed lease enforcement is implemented. Node-agent
|
|
release `0.2.230` is built, published under `/downloads`, registered as the
|
|
active `rap-node-agent` dev release, and deployed on docker-test
|
|
`test-1/2/3`; all three report `0.2.230`, healthy, and current after policy
|
|
update. When a cluster authority public key is pinned, the node-agent now
|
|
rejects unsigned `rap_fsc_*` service-channel requests and requires the
|
|
signed `rap.fabric_service_channel_lease_authority.v1` payload/signature
|
|
headers. Legacy unsigned tokens remain accepted only in unpinned test mode.
|
|
Live smoke proved unsigned POST is rejected with 403 while signed lease POST
|
|
is accepted with 202:
|
|
`artifacts/c18z47-service-channel-signed-lease-enforcement-smoke-result.json`.
|
|
- C18Z48 service-channel backend introspection compatibility is implemented.
|
|
Backend `rap-backend:fabric-service-channel-0.2.231` is built/deployed on
|
|
docker-test. Node-agent/host-agent artifacts `0.2.232` are published under
|
|
`/downloads`; `rap-node-agent` release `0.2.232` is registered and deployed
|
|
on `test-1/2/3`, and all three report healthy/current. When signed
|
|
service-channel authority headers are absent but cluster authority is pinned,
|
|
node-agent now calls backend lease introspection before accepting an unsigned
|
|
token. Bad tokens are still rejected. Live smoke passed:
|
|
`artifacts/c18z48-service-channel-introspection-smoke-result.json`.
|
|
- C18Z49 service-channel acceptance telemetry is implemented in node-agent
|
|
`0.2.232`. Each accepted Fabric Service Channel ingress records
|
|
`accepted_by=signed|introspection|legacy_unsigned`, route preference, and
|
|
backend-fallback state in structured node logs. HTTP packet ingress also
|
|
returns `X-RAP-Service-Channel-Accepted-By` for smoke/diagnostics.
|
|
- C18Z50 durable service-channel lease introspection is implemented. Migration
|
|
`000029_fabric_service_channel_leases` adds a durable lease table keyed by
|
|
cluster/channel and stores only `token_hash` plus a scrubbed lease payload
|
|
with the raw bearer token removed. Backend
|
|
`rap-backend:fabric-service-channel-0.2.233` is built/deployed on
|
|
docker-test after applying the migration. Introspection now reads memory
|
|
first, then durable storage, so compatibility clients survive backend
|
|
restart. Live smoke restarted `rap_test_backend`, accepted the unsigned token
|
|
through introspection, rejected a bad token, and verified the durable lease
|
|
omits the raw token:
|
|
`artifacts/c18z50-service-channel-durable-introspection-smoke-result.json`.
|
|
- C18Z51 service-channel lease maintenance is implemented. Backend
|
|
`rap-backend:fabric-service-channel-0.2.234` is built/deployed on
|
|
docker-test. New endpoints list durable service-channel lease maintenance
|
|
state and run bounded expired-lease cleanup:
|
|
`GET /clusters/{clusterID}/fabric/service-channels/leases` and
|
|
`POST /clusters/{clusterID}/fabric/service-channels/leases/cleanup`.
|
|
Web-admin adds a `Service-channel leases` panel with active/expired counts,
|
|
recent lease rows, and cleanup action. Live smoke issued a 1-second lease,
|
|
observed it as expired, cleaned it up, and verified it disappeared:
|
|
`artifacts/c18z51-service-channel-lease-maintenance-smoke-result.json`.
|
|
- C18Z52 service-channel access telemetry visibility is implemented. Backend
|
|
`rap-backend:fabric-service-channel-0.2.235` is built/deployed on
|
|
docker-test; node-agent/host-agent `0.2.235` artifacts are published under
|
|
`/downloads`, registered as active dev releases, and deployed on
|
|
`test-1/2/3`. Node-agent now reports accepted service-channel ingress
|
|
counters by `signed`, `introspection`, and `legacy_unsigned`, including
|
|
backend-fallback count and last accepted timestamp. Backend exposes
|
|
`GET /clusters/{clusterID}/fabric/service-channels/access-telemetry`,
|
|
reading telemetry observations with heartbeat metadata fallback. Web-admin
|
|
adds a `Service-channel access` panel with cluster totals and per-node rows.
|
|
Live smoke sent packets through test-1, observed
|
|
`X-RAP-Service-Channel-Accepted-By: introspection`, and verified backend
|
|
aggregate visibility:
|
|
`artifacts/c18z52-service-channel-access-telemetry-smoke-result.json`.
|
|
- C18Z53 service-channel access/session correlation is implemented. Backend
|
|
`rap-backend:fabric-service-channel-0.2.236` is built/deployed on
|
|
docker-test; node-agent remains `0.2.235`. The access telemetry endpoint now
|
|
correlates accepted ingress counters with active durable service-channel
|
|
leases, selected entry/exit nodes, primary route status, explicit backend
|
|
fallback, and latest route-quality feedback when a route exists. Web-admin's
|
|
`Service-channel access` panel now shows active channel rows before per-node
|
|
counters, so operators can see whether a live service channel is using normal
|
|
route quality feedback or degraded backend fallback. Live smoke created an
|
|
active lease, sent ingress traffic through test-1, and verified active
|
|
channel correlation plus fallback visibility:
|
|
`artifacts/c18z53-service-channel-access-correlation-smoke-result.json`.
|
|
- C18Z54 normal-route access correlation is smoke-proven on the existing
|
|
C18Z53 backend/admin surface. New smoke creates a temporary direct
|
|
`vpn_packets` route intent, injects healthy route-quality heartbeat
|
|
telemetry, issues a service-channel lease that selects the normal primary
|
|
route, sends ingress traffic, and verifies the access telemetry active
|
|
channel row is `ready`, not backend fallback, with `route_feedback_status`
|
|
`healthy`, rolling quality counters, and last send duration:
|
|
`artifacts/c18z54-service-channel-normal-route-access-smoke-result.json`.
|
|
- C18Z55 degraded normal-route access correlation is smoke-proven on the same
|
|
backend/admin surface. The smoke first issues a lease on a normal primary
|
|
`vpn_packets` route, then injects degraded/fenced route-quality heartbeat
|
|
feedback for that already-selected route. Access telemetry correctly reports
|
|
the active channel as `ready` and `force_backend_fallback=false`, while route
|
|
feedback is `fenced`, rolling failure/drop/slow counters are visible, and the
|
|
aggregate access status becomes `degraded` because `degraded_route_count > 0`:
|
|
`artifacts/c18z55-service-channel-degraded-route-access-smoke-result.json`.
|
|
- C18Z56 active-channel remediation diagnostics are implemented. Backend
|
|
`rap-backend:fabric-service-channel-0.2.237` is built/deployed on
|
|
docker-test; node-agent remains `0.2.235`. Active access telemetry channel
|
|
rows now include `remediation_action`, `remediation_reason`,
|
|
`remediation_route_id`, `remediation_route_status`, and an operator hint.
|
|
Decisions distinguish explicit backend fallback, degraded/fenced normal
|
|
route with an authorized alternate (`prefer_alternate_route`), degraded/fenced
|
|
route needing rebuild (`rebuild_route`), and healthy route (`none`).
|
|
Web-admin shows the remediation action in the `Service-channel access`
|
|
active-channel table. C18Z55 smoke now verifies
|
|
`remediation_action=rebuild_route`; backend unit coverage verifies the
|
|
alternate-route remediation branch.
|
|
- C18Z56 alternate-route remediation is also live-smoke-proven. New smoke
|
|
creates primary and authorized alternate `vpn_packets` routes, issues a lease
|
|
while primary is still healthy/selected, then injects fenced feedback for the
|
|
selected primary. Access telemetry keeps the active channel on the normal
|
|
route with `force_backend_fallback=false`, reports `route_feedback_status`
|
|
`fenced`, and recommends `remediation_action=prefer_alternate_route` with the
|
|
alternate route id/status; `degraded_fallback_channel_count` stays zero:
|
|
`artifacts/c18z56-service-channel-alternate-remediation-smoke-result.json`.
|
|
- C18Z57 bounded remediation command contract is implemented. Backend
|
|
`rap-backend:fabric-service-channel-0.2.238` is built/deployed on
|
|
docker-test; node-agent remains `0.2.235`. Active access telemetry channel
|
|
rows now include `remediation_command` for non-noop remediation actions, with
|
|
schema version, deterministic command id, action, channel/resource/service,
|
|
entry/exit, primary route, replacement route when present, reason/operator
|
|
hint, issued time, and a bounded TTL capped to the lease lifetime. Web-admin
|
|
marks remediation rows with `cmd` when this machine-readable command is
|
|
present. Live smoke proves a fenced selected primary route with an authorized
|
|
alternate emits a `prefer_alternate_route` command pointing at the alternate:
|
|
`artifacts/c18z57-service-channel-remediation-command-smoke-result.json`.
|
|
- C18Z58 service-channel remediation command consumption is implemented.
|
|
Backend `rap-backend:fabric-service-channel-0.2.239` and node-agent
|
|
`rap-node-agent:0.2.237` are built/deployed on docker-test (`test-1/2/3`).
|
|
Backend now projects active `remediation_command` items into node-scoped
|
|
synthetic mesh config as `service_channel_remediation_commands`. Node-agent
|
|
parses those commands and turns `prefer_alternate_route` into an explicit
|
|
route-manager `applied` decision with source
|
|
`service_channel_remediation_command`, so an active channel that still
|
|
presents the old primary route can be routed through the replacement route.
|
|
Web-admin node details show remediation-command count/table in the Mesh tab.
|
|
Live smoke proves access telemetry, synthetic config projection, and
|
|
node-agent route-manager consumption:
|
|
`artifacts/c18z58-service-channel-remediation-apply-smoke-result.json`.
|
|
- C18Z59 active remediation traffic proof is smoke-proven on the same
|
|
backend/node-agent images with production forwarding enabled on docker-test
|
|
`test-1/2/3`. The smoke sends service-channel traffic before/after the
|
|
remediation command is consumed, then verifies runtime heartbeat evidence:
|
|
`last_selected_route_id` and flow-scheduler `last_route_id` move to the
|
|
replacement route, `send_successes=1`, `send_failures=0`,
|
|
`send_fallback_local=0`, and no degraded backend fallback is recommended.
|
|
Result:
|
|
`artifacts/c18z59-service-channel-remediation-traffic-smoke-result.json`.
|
|
- C18Z60 multi-flow remediation traffic proof is smoke-proven. The smoke sends
|
|
a batch of twelve IPv4/TCP-like packets that classify into multiple
|
|
independent VPN flow channels after the remediation command is consumed.
|
|
Runtime heartbeat evidence shows the replacement route selected, at least two
|
|
flow-scheduler channels on that route, no local/backend fallback, no flow
|
|
drops, and no route send failures. Result:
|
|
`artifacts/c18z60-service-channel-remediation-multiflow-smoke-result.json`.
|
|
- C18Z61 pressure remediation traffic proof is smoke-proven. The smoke sends a
|
|
batch of 128 IPv4/TCP-like packets after remediation; runtime evidence shows
|
|
32 replacement-route flow stats, scheduler high-watermark 5,
|
|
max-in-flight 4, `send_fallback_local=0`, route failures 0, and flow/scheduler
|
|
drops 0. Result:
|
|
`artifacts/c18z61-service-channel-remediation-pressure-smoke-result.json`.
|
|
- C18Z62 service-channel QoS class wiring is implemented in node-agent and
|
|
live-smoke-proven on docker-test image `rap-node-agent:0.2.238-c18z62`.
|
|
Service-channel HTTP ingress accepts neutral `X-RAP-Traffic-Class`
|
|
(`control`, `interactive`, `reliable`, `bulk`, `droppable`) and the flow
|
|
scheduler keeps distinct traffic-class channel ids/stats while preserving the
|
|
old default bulk channel ids. Unit tests prove priority ordering
|
|
`control > interactive > reliable > bulk > droppable`; live smoke proves a
|
|
bulk 128-packet pressure batch plus an interactive packet both move through
|
|
the remediation replacement route with no local/backend fallback, drops, or
|
|
route failures. Result:
|
|
`artifacts/c18z62-service-channel-remediation-qos-smoke-result.json`.
|
|
- C18Z63 concurrent QoS isolation is implemented and unit-proven. A controlled
|
|
runtime test holds a bulk traffic-class send in-flight with a blocking
|
|
production transport, then sends an independent interactive traffic-class
|
|
packet through the same ingress; the interactive send completes before the
|
|
bulk release, with `MaxInFlight >= 2`, traffic-class-specific stats, no drops,
|
|
and no failures. This proves the shared Fabric Service Channel runtime does
|
|
not globally serialize interactive/control-style traffic behind bulk work.
|
|
Artifact:
|
|
`artifacts/c18z63-service-channel-concurrent-qos-go-test.jsonl`.
|
|
- C18Z64 traffic-class telemetry aggregation is implemented and live-proven on
|
|
docker-test image `rap-node-agent:0.2.239-c18z64`. `rap.fabric_flow_scheduler.v1`
|
|
snapshots now include `traffic_class_counts`, giving backend/admin/diagnostics
|
|
a compact count of active flow channels per traffic class without scanning
|
|
every channel stat. Unit coverage proves the counts for explicit
|
|
control/interactive/bulk classes and for the concurrent bulk+interactive
|
|
isolation case. Live smoke re-ran the QoS path on `test-1/2/3`; latest
|
|
heartbeat snapshot showed `traffic_class_counts` `bulk=32`,
|
|
`interactive=12`, drops 0. Artifacts:
|
|
`artifacts/c18z64-service-channel-traffic-class-telemetry-go-test.jsonl`,
|
|
`artifacts/c18z64-service-channel-traffic-class-telemetry-live-smoke-result.json`,
|
|
and
|
|
`artifacts/c18z64-service-channel-traffic-class-telemetry-live-snapshot.json`.
|
|
- C18Z65/C18Z66 backend/admin QoS diagnostics are implemented and live-proven.
|
|
Backend `rap-backend:fabric-service-channel-0.2.241-c18z66` is deployed on
|
|
docker-test and projects runtime `traffic_class_counts`, flow channel count,
|
|
max in-flight, dropped, and high-watermark from node heartbeats into
|
|
`GET /fabric/service-channels/access-telemetry` at node, active-channel, and
|
|
cluster aggregate levels. Web-admin Service-channel access shows flow QoS
|
|
chips/rows for cluster totals, active channels, and nodes. Live API aggregate
|
|
result showed `bulk=32`, `interactive=12`, `flow_channel_count=44`,
|
|
`flow_max_in_flight=4`. Artifacts:
|
|
`artifacts/c18z65-service-channel-access-qos-telemetry-api-result.json`,
|
|
`artifacts/c18z65-service-channel-access-qos-telemetry-smoke-result.json`,
|
|
and
|
|
`artifacts/c18z66-service-channel-access-qos-aggregate-api-result.json`.
|
|
- C18Z67 live concurrent QoS proof is implemented and smoke-proven against
|
|
docker-test backend `rap-backend:fabric-service-channel-0.2.241-c18z66` and
|
|
node-agent image `rap-node-agent:0.2.239-c18z64`. The smoke pushes six
|
|
parallel bulk service-channel HTTP packet requests while an interactive
|
|
traffic-class request is injected through the same entry path after
|
|
remediation. Run `c18z67-20260508-213452` accepted all 6 bulk requests,
|
|
forwarded 3072 post-remediation packets, completed the interactive request in
|
|
132 ms, observed 32 bulk and 12 interactive replacement-route flow stats, and
|
|
kept local/backend fallback, route failures, flow drops, and scheduler drops
|
|
at 0. Artifact:
|
|
`artifacts/c18z67-service-channel-concurrent-qos-live-smoke-result.json`.
|
|
- C18Z68 service-channel flow-health guard is implemented and deployed on
|
|
docker-test as `rap-backend:fabric-service-channel-0.2.242-c18z68`, with
|
|
web-admin rebuilt/deployed. Access telemetry now projects
|
|
`flow_health_status` and `flow_health_reason` at cluster, node, and
|
|
active-channel levels from traffic-class counts, queue pressure, flow drops,
|
|
backend fallback, route-quality failures/drops/slow samples, and route send
|
|
latency. Web-admin shows explicit flow-health chips beside flow QoS so
|
|
sustained bulk pressure, degraded latency, fallback, and drops are visible
|
|
before adding user services. Verification passed:
|
|
`go test ./internal/modules/cluster`, web-admin `npm run build`, updated
|
|
C18Z67 live smoke against backend `0.2.242-c18z68`, and live API artifact
|
|
`artifacts/c18z68-service-channel-flow-health-api-result.json`.
|
|
- C18Z69 node-side adaptive backpressure is implemented and deployed on
|
|
docker-test image `rap-node-agent:0.2.243-c18z69` for `test-1/2/3`.
|
|
`FabricFlowScheduler` now calculates per-traffic-class
|
|
`recommended_parallel_windows` and reports `adaptive_backpressure_active` /
|
|
`adaptive_backpressure_reason` in runtime heartbeat snapshots. Bulk and
|
|
droppable classes are reduced first under pressure, reliable is reduced
|
|
moderately, while control/interactive keep their full window unless their own
|
|
class has drops/failures/slow samples. Live C18Z69 smoke wraps the C18Z67
|
|
pressure path and verified `bulk=1`, `droppable=1`, `reliable=3`,
|
|
`interactive=4`, `control=4`, `bulk=32`, `interactive=12`, high-watermark
|
|
72, max-in-flight 4, drops 0, and
|
|
`bulk_window_reduced_to_protect_interactive`. Artifacts:
|
|
`artifacts/c18z67-service-channel-concurrent-qos-live-smoke-result.json` and
|
|
`artifacts/c18z69-service-channel-adaptive-backpressure-smoke-result.json`.
|
|
- C18Z70 backend/admin adaptive backpressure visibility is implemented and
|
|
deployed on docker-test as
|
|
`rap-backend:fabric-service-channel-0.2.244-c18z70`; web-admin is rebuilt and
|
|
deployed. Access telemetry now projects node-agent
|
|
`recommended_parallel_windows`, `adaptive_backpressure_active`, and
|
|
`adaptive_backpressure_reason` at cluster, node, and active-channel levels.
|
|
Cluster aggregation uses the minimum non-zero recommended window per class,
|
|
so the operator sees the most conservative active runtime limit. Web-admin
|
|
shows adaptive windows next to flow health and flow QoS. Live API returned
|
|
`adaptive=true`, reason `bulk_window_reduced_to_protect_interactive`, and
|
|
windows `bulk=1`, `droppable=1`, `reliable=3`, `interactive=4`,
|
|
`control=4`. Verification passed: `go test ./internal/modules/cluster`,
|
|
web-admin `npm run build`, C18Z69 live smoke, and
|
|
`artifacts/c18z70-service-channel-adaptive-telemetry-api-result.json`.
|
|
- C18Z71 adaptive policy contract is implemented and deployed on docker-test as
|
|
`rap-backend:fabric-service-channel-0.2.245-c18z71` with node-agent image
|
|
`rap-node-agent:0.2.245-c18z71` on `test-1/2/3`. Backend exposes audited
|
|
`GET/PUT /clusters/{clusterID}/fabric/service-channels/adaptive-policy` for
|
|
max parallel window, queue/bulk pressure thresholds, and per-class windows.
|
|
The effective policy is embedded in signed node synthetic config and
|
|
node-agent runtime heartbeat snapshots now report
|
|
`adaptive_policy_fingerprint`. The scheduler consumes the policy at runtime:
|
|
default policy preserves the C18Z69 behavior, while the C18Z71 live smoke
|
|
proved an operator policy can raise max window to 6 and bulk pressure window
|
|
to 2 while keeping interactive/control at 6. During smoke, a signed synthetic
|
|
config hash mismatch was found and fixed by preserving adaptive policy
|
|
provenance fields in the node-agent client model. Verification passed:
|
|
`go test ./internal/modules/cluster`,
|
|
`go test ./cmd/rap-node-agent ./internal/mesh ./internal/vpnruntime ./internal/client ./internal/config`,
|
|
web-admin `npm run build`, C18Z71 live smoke, and C18Z69 regression smoke.
|
|
Artifacts:
|
|
`artifacts/c18z71-service-channel-adaptive-policy-smoke-result.json` and
|
|
`artifacts/c18z69-service-channel-adaptive-backpressure-smoke-result.json`.
|
|
- C18Z72 service-channel pool/failover policy contract is implemented and
|
|
deployed on docker-test as
|
|
`rap-backend:fabric-service-channel-0.2.246-c18z72`; node-agent remains
|
|
`rap-node-agent:0.2.245-c18z71` on `test-1/2/3`. Backend exposes audited
|
|
`GET/PUT /clusters/{clusterID}/fabric/service-channels/pool-policy` for
|
|
entry/exit pool constraints, preferred entry/exit, selection strategy,
|
|
route/entry/exit failover modes, backend fallback allowance, and sticky
|
|
session mode. Lease issuance now applies the effective policy before route
|
|
selection, constrains `entry_pool`/`exit_pool`, chooses policy preferred
|
|
nodes when present, embeds `pool_policy` provenance in the lease, and signs
|
|
it into `rap.fabric_service_channel_lease_authority.v1`. Web-admin API/types
|
|
know the new policy contract. Verification passed:
|
|
`go test ./internal/modules/cluster`, web-admin `npm run build`,
|
|
C18Z72 live smoke, and C18Z71 regression smoke. Artifact:
|
|
`artifacts/c18z72-service-channel-pool-policy-smoke-result.json`.
|
|
- C18Z73 pool-policy remediation guard and telemetry is implemented and
|
|
deployed on docker-test as
|
|
`rap-backend:fabric-service-channel-0.2.247-c18z73` with node-agent image
|
|
`rap-node-agent:0.2.247-c18z73` on `test-1/2/3`; web-admin is rebuilt and
|
|
deployed. Active access telemetry now projects the signed
|
|
`pool_policy_fingerprint`, remediation guard status/reason, and guarded
|
|
remediation commands. Backend remediation rejects an alternate route outside
|
|
the signed entry/exit lease pools and emits `rebuild_route` instead of
|
|
`prefer_alternate_route`; node-agent defensively ignores guarded rejected
|
|
remediation commands before route-manager application. Web-admin shows guard
|
|
chips in access telemetry and node synthetic-config remediation rows.
|
|
Verification passed:
|
|
`go test ./internal/modules/cluster ./internal/platform/runtime ./internal/modules/nodeagent`,
|
|
`go test ./cmd/rap-node-agent ./internal/mesh ./internal/vpnruntime ./internal/config`,
|
|
web-admin `npm run build`, C18Z73 live smoke, C18Z72 regression smoke, and
|
|
C18Z71/C18Z67 live regression smoke. Artifacts:
|
|
`artifacts/c18z73-service-channel-pool-policy-remediation-guard-smoke-result.json`,
|
|
`artifacts/c18z72-service-channel-pool-policy-smoke-result.json`,
|
|
`artifacts/c18z71-service-channel-adaptive-policy-smoke-result.json`, and
|
|
`artifacts/c18z67-service-channel-concurrent-qos-live-smoke-result.json`.
|
|
- C18Z74 service-channel remediation execution visibility is implemented and
|
|
deployed on docker-test as
|
|
`rap-backend:fabric-service-channel-0.2.248-c18z74` with node-agent image
|
|
`rap-node-agent:0.2.248-c18z74` on `test-1/2/3`; web-admin is rebuilt and
|
|
deployed. Active access telemetry now computes
|
|
`remediation_execution_status`, reason, generation, and observed timestamp
|
|
by correlating active remediation commands with the entry node's latest
|
|
route-manager heartbeat. `prefer_alternate_route` commands show
|
|
`waiting_node_apply` until the node reports a matching route-manager decision
|
|
and then `applied`; guarded commands show `rejected_by_policy_guard`; bounded
|
|
`rebuild_route` commands show `pending_rebuild_request`. The execution state
|
|
is copied into the machine-readable remediation command and displayed in
|
|
web-admin access telemetry / node synthetic remediation rows. Verification
|
|
passed:
|
|
`go test ./internal/modules/cluster ./internal/platform/runtime ./internal/modules/nodeagent`,
|
|
`go test ./cmd/rap-node-agent ./internal/mesh ./internal/vpnruntime ./internal/config`,
|
|
web-admin `npm run build`, C18Z74 live smoke, C18Z73 regression smoke, and
|
|
C18Z72 regression smoke. Artifacts:
|
|
`artifacts/c18z74-service-channel-remediation-execution-smoke-result.json`,
|
|
`artifacts/c18z67-service-channel-concurrent-qos-live-smoke-result.json`,
|
|
`artifacts/c18z73-service-channel-pool-policy-remediation-guard-smoke-result.json`,
|
|
and `artifacts/c18z72-service-channel-pool-policy-smoke-result.json`.
|
|
- C18Z75 durable remediation rebuild intent foundation is implemented and
|
|
deployed on docker-test as
|
|
`rap-backend:fabric-service-channel-0.2.249-c18z75`; node-agent remains
|
|
`rap-node-agent:0.2.248-c18z74` on `test-1/2/3`. When a node fetches
|
|
synthetic config containing a `rebuild_route` remediation command, backend
|
|
now records a durable row in the existing
|
|
`fabric_service_channel_route_rebuild_attempts` ledger with
|
|
`rebuild_status=requested` / `outcome=rebuild_requested`, or
|
|
`rebuild_status=rejected` / `outcome=policy_guard_rejected` when the pool
|
|
policy guard rejects it. Access telemetry correlates that ledger row back to
|
|
the active channel and reports `rebuild_request_recorded` or
|
|
`rebuild_request_rejected` in `remediation_execution_status`. The C18Z75
|
|
smoke isolates a route pair, proves `rebuild_route`, fetches synthetic
|
|
config to persist the intent, verifies the rebuild ledger row, and verifies
|
|
access telemetry reports the recorded execution state. Verification passed:
|
|
`go test ./internal/modules/cluster ./internal/platform/runtime ./internal/modules/nodeagent`,
|
|
`go test ./cmd/rap-node-agent ./internal/mesh ./internal/vpnruntime ./internal/config`,
|
|
web-admin `npm run build`, C18Z75 live smoke, C18Z73 regression smoke, and
|
|
C18Z72 regression smoke. Artifacts:
|
|
`artifacts/c18z75-service-channel-rebuild-intent-smoke-result.json`,
|
|
`artifacts/c18z73-service-channel-pool-policy-remediation-guard-smoke-result.json`,
|
|
and `artifacts/c18z72-service-channel-pool-policy-smoke-result.json`.
|
|
- C18Z76 service-channel rebuild-route node acknowledgement is implemented and
|
|
deployed on docker-test as
|
|
`rap-backend:fabric-service-channel-0.2.250-c18z76` with node-agent image
|
|
`rap-node-agent:0.2.250-c18z76` on `test-1/2/3`. Node-agent now consumes
|
|
allowed `rebuild_route` remediation commands as route-manager decisions with
|
|
`rebuild_status=pending_degraded_fallback` and
|
|
`decision_source=service_channel_remediation_command`; guarded commands are
|
|
still ignored. Backend access telemetry correlates this route-manager
|
|
acknowledgement with the durable ledger intent and reports
|
|
`rebuild_request_recorded_node_pending`. Verification passed:
|
|
`go test ./internal/modules/cluster ./internal/platform/runtime ./internal/modules/nodeagent`,
|
|
`go test ./cmd/rap-node-agent ./internal/agent ./internal/mesh ./internal/vpnruntime ./internal/config`,
|
|
C18Z76 live smoke, C18Z75 regression smoke, and C18Z74/C18Z67 regression
|
|
smoke. Artifacts:
|
|
`artifacts/c18z76-service-channel-rebuild-node-pending-smoke-result.json`,
|
|
`artifacts/c18z75-service-channel-rebuild-intent-smoke-result.json`,
|
|
`artifacts/c18z74-service-channel-remediation-execution-smoke-result.json`,
|
|
and `artifacts/c18z67-service-channel-concurrent-qos-live-smoke-result.json`.
|
|
- C18Z77 service-channel rebuild planner resolution is implemented and
|
|
deployed on docker-test as
|
|
`rap-backend:fabric-service-channel-0.2.251-c18z77` with node-agent image
|
|
`rap-node-agent:0.2.251-c18z77` on `test-1/2/3`. Backend now resolves
|
|
durable `rebuild_route` remediation requests during node-scoped synthetic
|
|
config generation: it keeps lease pool-policy guardrails, records
|
|
`applied` / `replacement_selected` when a signed-pool-valid alternate route
|
|
exists, records `no_alternate` when no safe alternate exists, records
|
|
`deferred_by_policy` when the active lease cannot authorize the replacement,
|
|
and records `expired` for stale commands. When a replacement is applied, the
|
|
same command id is projected as a route-manager decision so node-agent can
|
|
consume the resolved planner decision without duplicating the raw command.
|
|
Access telemetry reports planner states such as `rebuild_request_applied`
|
|
and `rebuild_request_no_alternate`. Verification passed:
|
|
`go test ./internal/modules/cluster ./internal/platform/runtime ./internal/modules/nodeagent`,
|
|
`go test ./cmd/rap-node-agent ./internal/agent ./internal/mesh ./internal/vpnruntime ./internal/config`,
|
|
C18Z77 live smoke, C18Z75 regression smoke, and C18Z74/C18Z67 regression
|
|
smoke. Artifacts:
|
|
`artifacts/c18z77-service-channel-rebuild-planner-resolution-smoke-result.json`,
|
|
`artifacts/c18z75-service-channel-rebuild-intent-smoke-result.json`,
|
|
`artifacts/c18z74-service-channel-remediation-execution-smoke-result.json`,
|
|
and `artifacts/c18z67-service-channel-concurrent-qos-live-smoke-result.json`.
|
|
- C18Z78 service-channel rebuild planner applied-branch visibility is
|
|
implemented and deployed on docker-test as
|
|
`rap-backend:fabric-service-channel-0.2.252-c18z78` with node-agent image
|
|
`rap-node-agent:0.2.252-c18z78` on `test-1/2/3`; web-admin is rebuilt and
|
|
deployed to `rap_web_admin`. The admin access-telemetry execution column and
|
|
node synthetic remediation rows now render planner outcomes with explicit
|
|
labels and tones: `rebuild_request_applied` is good,
|
|
`rebuild_request_recorded(_node_pending)`, `rebuild_request_no_alternate`,
|
|
and `rebuild_request_deferred_by_policy` are warning states, while rejected
|
|
or expired requests are bad states. The C18Z78 live smoke proves the applied
|
|
planner branch: a primary route is leased first, the primary route is then
|
|
degraded, an alternate route is added after the lease, synthetic config
|
|
fetch resolves the existing `rebuild_route` command to `applied` /
|
|
`replacement_selected`, and access telemetry reports
|
|
`rebuild_request_applied`. Verification passed:
|
|
`go test ./internal/modules/cluster ./internal/platform/runtime ./internal/modules/nodeagent`,
|
|
`go test ./cmd/rap-node-agent ./internal/agent ./internal/mesh ./internal/vpnruntime ./internal/config`,
|
|
web-admin `npm run build`, C18Z78 live smoke, C18Z77 regression smoke, and
|
|
C18Z74/C18Z67 regression smoke. Artifacts:
|
|
`artifacts/c18z78-service-channel-rebuild-planner-applied-smoke-result.json`,
|
|
`artifacts/c18z77-service-channel-rebuild-planner-resolution-smoke-result.json`,
|
|
`artifacts/c18z74-service-channel-remediation-execution-smoke-result.json`,
|
|
and `artifacts/c18z67-service-channel-concurrent-qos-live-smoke-result.json`.
|
|
- C18Z79 service-channel planner-to-runtime loop proof is implemented and
|
|
deployed on docker-test as
|
|
`rap-backend:fabric-service-channel-0.2.253-c18z79` with node-agent image
|
|
`rap-node-agent:0.2.253-c18z79` on `test-1/2/3`. The new live smoke extends
|
|
the C18Z78 applied branch: after planner resolves the existing
|
|
`rebuild_route` command to `applied` / `replacement_selected`, the entry node
|
|
reports a route-manager decision for the same `rebuild_request_id`, reports
|
|
transition `applied_rebuild`, and live service-channel packet ingress selects
|
|
the replacement route with no local/backend fallback, route failures, or flow
|
|
drops. Verification passed:
|
|
`go test ./internal/modules/cluster ./internal/platform/runtime ./internal/modules/nodeagent`,
|
|
`go test ./cmd/rap-node-agent ./internal/agent ./internal/mesh ./internal/vpnruntime ./internal/config`,
|
|
C18Z79 live smoke, C18Z78 and C18Z77 sequential regressions, and C18Z67
|
|
concurrent QoS regression. Artifact:
|
|
`artifacts/c18z79-service-channel-planner-runtime-loop-smoke-result.json`.
|
|
- C18Z80 service-channel sustained post-rebuild pressure proof is implemented
|
|
and deployed on docker-test as
|
|
`rap-backend:fabric-service-channel-0.2.254-c18z80` with node-agent image
|
|
`rap-node-agent:0.2.254-c18z80` on `test-1/2/3`. The new live smoke keeps the
|
|
C18Z79 planner-applied loop, then sends five post-rebuild bursts of mixed
|
|
`interactive`, `bulk`, and `reliable` VPN packet batches. It proves every
|
|
burst is accepted by the service-channel runtime, every burst reports the
|
|
replacement route, the stale primary is not reselected, and fallback,
|
|
route-failure, flow-drop, and scheduler-drop deltas stay zero from the
|
|
pre-pressure baseline. Smoke route hygiene was tightened: C18Z67 now disables
|
|
pre-existing active `vpn_packets` intents for its entry/exit pair, and
|
|
C18Z79/C18Z80 expire their temporary primary/alternate intents after a
|
|
successful run. Verification passed:
|
|
`go test ./internal/modules/cluster ./internal/platform/runtime ./internal/modules/nodeagent`,
|
|
`go test ./cmd/rap-node-agent ./internal/agent ./internal/mesh ./internal/vpnruntime ./internal/config`,
|
|
C18Z80 live smoke, C18Z79 regression smoke, and C18Z67 concurrent QoS
|
|
regression. Artifact:
|
|
`artifacts/c18z80-service-channel-post-rebuild-pressure-smoke-result.json`.
|
|
- C18Z81 service-channel replacement-degradation recovery proof is implemented
|
|
and deployed on docker-test as
|
|
`rap-backend:fabric-service-channel-0.2.255-c18z81` with node-agent image
|
|
`rap-node-agent:0.2.255-c18z81` on `test-1/2/3`. The new live smoke proves
|
|
the negative branch after C18Z80: once the initial replacement is applied and
|
|
used, a generation-valid fenced feedback report for that replacement causes
|
|
the Control Plane to select a new safe recovery route. Live traffic then
|
|
moves to the recovery route, the degraded replacement is not reselected, and
|
|
fallback, route-failure, flow-drop, and scheduler-drop deltas stay zero for
|
|
the recovery send. The smoke also documents an important guardrail: stale
|
|
route-generation feedback must not trigger recovery. C18Z67/C18Z79 were
|
|
tightened to check per-run counter deltas rather than cumulative runtime
|
|
counters. Verification passed:
|
|
`go test ./internal/modules/cluster ./internal/platform/runtime ./internal/modules/nodeagent`,
|
|
`go test ./cmd/rap-node-agent ./internal/agent ./internal/mesh ./internal/vpnruntime ./internal/config`,
|
|
C18Z81 live smoke, C18Z80 regression smoke, C18Z79 regression smoke, and
|
|
C18Z67 concurrent QoS regression. Artifact:
|
|
`artifacts/c18z81-service-channel-replacement-degradation-recovery-smoke-result.json`.
|
|
- C18Z82 service-channel no-safe-recovery proof is implemented and deployed on
|
|
docker-test as `rap-backend:fabric-service-channel-0.2.256-c18z82` with
|
|
node-agent image `rap-node-agent:0.2.256-c18z82` on `test-1/2/3`. The new
|
|
live smoke proves the branch where the original primary is degraded, the
|
|
replacement is applied and used, then that replacement reports
|
|
generation-valid fenced feedback while no new safe recovery route exists.
|
|
Node-scoped synthetic config reports
|
|
`service_channel_feedback_no_alternate` with
|
|
`pending_degraded_fallback`; score reasons include
|
|
`no_unfenced_alternate_route` and
|
|
`backend_relay_degraded_fallback_until_rebuild`, so the Control Plane exposes
|
|
an explicit degraded/no-alternate state instead of silently sticking to a bad
|
|
replacement. Verification passed:
|
|
`go test ./internal/modules/cluster ./internal/platform/runtime ./internal/modules/nodeagent`,
|
|
`go test ./cmd/rap-node-agent ./internal/agent ./internal/mesh ./internal/vpnruntime ./internal/config`,
|
|
C18Z82 live smoke, C18Z81 recovery regression, C18Z80 pressure regression,
|
|
and C18Z67 concurrent QoS regression. Artifact:
|
|
`artifacts/c18z82-service-channel-no-safe-recovery-smoke-result.json`.
|
|
- C18Z83 service-channel access-telemetry no-safe projection is implemented and
|
|
deployed on docker-test as `rap-backend:fabric-service-channel-0.2.257-c18z83`;
|
|
node-agent remains `rap-node-agent:0.2.256-c18z82` on `test-1/2/3`, and
|
|
web-admin is rebuilt/deployed to `rap_web_admin`. Active access telemetry
|
|
channels now expose route-decision source, route id, replacement route id,
|
|
rebuild status/reason/generation, and score reasons. Web-admin shows a
|
|
dedicated `decision` column in the active-channel table. The live smoke
|
|
proves no-safe recovery is visible through access telemetry as
|
|
`service_channel_feedback_no_alternate` /
|
|
`pending_degraded_fallback`, while durable ledger state can still report
|
|
`rebuild_request_no_alternate`. Verification passed:
|
|
`go test ./internal/modules/cluster ./internal/platform/runtime ./internal/modules/nodeagent`,
|
|
web-admin `npm run build`, and C18Z83 live smoke. Artifact:
|
|
`artifacts/c18z83-service-channel-access-telemetry-no-safe-smoke-result.json`.
|
|
- C18Z84 service-channel access-decision aggregate proof is implemented and
|
|
deployed on docker-test as `rap-backend:fabric-service-channel-0.2.258-c18z84`;
|
|
node-agent remains `rap-node-agent:0.2.256-c18z82` on `test-1/2/3`, and
|
|
web-admin is rebuilt/deployed to `rap_web_admin`. Access telemetry now
|
|
exposes aggregate route-decision counters:
|
|
`route_decision_channel_count`, `replacement_decision_count`,
|
|
`applied_rebuild_decision_count`, `recovery_decision_count`, and
|
|
`no_safe_recovery_decision_count`. Web-admin summary chips show these counts,
|
|
and no-safe route decisions now prioritize the aggregate reason
|
|
`active_channels_no_safe_recovery` over generic missing access-report noise.
|
|
Verification passed:
|
|
`go test ./internal/modules/cluster ./internal/platform/runtime ./internal/modules/nodeagent`,
|
|
web-admin `npm run build`, C18Z84 live smoke, and C18Z83 regression smoke.
|
|
Artifact:
|
|
`artifacts/c18z84-service-channel-access-decision-aggregate-smoke-result.json`.
|
|
- C18Z85 service-channel access-decision incident projection is implemented and
|
|
deployed on docker-test as `rap-backend:fabric-service-channel-0.2.259-c18z85`;
|
|
node-agent remains `rap-node-agent:0.2.256-c18z82` on `test-1/2/3`, and
|
|
web-admin is rebuilt/deployed to `rap_web_admin`. Rebuild health summary now
|
|
carries access decision counts and prioritizes
|
|
`inspect_access_no_safe_recovery_route_pool_and_signed_policy` when no-safe
|
|
is active. Rebuild incidents now include `incident_source=access_decision`
|
|
entries with channel id and operator-facing severity/action, including
|
|
`access_no_safe_recovery` as a bad incident. Verification passed:
|
|
`go test ./internal/modules/cluster ./internal/platform/runtime ./internal/modules/nodeagent`,
|
|
web-admin `npm run build`, C18Z85 live smoke, and C18Z84 regression smoke.
|
|
Artifact:
|
|
`artifacts/c18z85-service-channel-access-decision-incident-smoke-result.json`.
|
|
- C18Z86 service-channel access-decision silence/acknowledgement is
|
|
implemented and deployed on docker-test as
|
|
`rap-backend:fabric-service-channel-0.2.261-c18z86`; node-agent remains
|
|
`rap-node-agent:0.2.256-c18z82` on `test-1/2/3`, and web-admin is
|
|
rebuilt/deployed to `rap_web_admin`. Rebuild alert silence requests now carry
|
|
`incident_source` and `channel_id`; `incident_source=access_decision`
|
|
no-safe incidents require `channel_id` and are stored with channel-scoped
|
|
route keys. Rebuild health and incident lists apply those silences, so an
|
|
acknowledged current-generation access no-safe incident is silenced and no
|
|
longer contributes to active bad count. Generation-change resurfacing is
|
|
covered in unit tests; live smoke proves the channel-scoped silence path.
|
|
Verification passed:
|
|
`go test ./internal/modules/cluster ./internal/platform/runtime ./internal/modules/nodeagent`,
|
|
web-admin `npm run build`, C18Z86 live smoke, and C18Z85 regression smoke.
|
|
Artifact:
|
|
`artifacts/c18z86-service-channel-access-decision-silence-smoke-result.json`.
|
|
- C18Z87 service-channel access-decision silence management is implemented and
|
|
deployed on docker-test as `rap-backend:fabric-service-channel-0.2.262-c18z87`;
|
|
node-agent remains `rap-node-agent:0.2.256-c18z82` on `test-1/2/3`, and
|
|
web-admin is rebuilt/deployed to `rap_web_admin`. Backend now exposes active
|
|
rebuild alert silences, enriches access-decision silences with
|
|
`incident_source`, `channel_id`, and `display_route_id`, and supports
|
|
unsilence by id. Web-admin shows an `Active rebuild silences` table with an
|
|
`unsilence` action. The live smoke proves the operator path:
|
|
access no-safe incident -> silence -> active silence listed -> unsilence ->
|
|
active bad incident restored. Verification passed:
|
|
`go test ./internal/modules/cluster ./internal/platform/runtime ./internal/modules/nodeagent`,
|
|
web-admin `npm run build`, C18Z87 live smoke, and C18Z86 regression smoke.
|
|
Artifact:
|
|
`artifacts/c18z87-service-channel-access-decision-unsilence-smoke-result.json`.
|
|
- C18Z88 service-channel access-decision resurface proof is implemented and
|
|
deployed on docker-test as `rap-backend:fabric-service-channel-0.2.263-c18z88`;
|
|
node-agent remains `rap-node-agent:0.2.256-c18z82` on `test-1/2/3`, and
|
|
web-admin is rebuilt/deployed to `rap_web_admin`. Access-decision incidents
|
|
now include resurface details (`alert_resurfaced_from_silence_id`,
|
|
`alert_resurfaced_previous_generation`, and
|
|
`alert_resurfaced_previous_until`) when a previously acknowledged
|
|
access-decision incident changes generation/route/channel and becomes active
|
|
again. Web-admin shows the previous generation/expiry beside resurfaced
|
|
incidents. The live smoke proves access no-safe -> silence current generation
|
|
-> route-decision generation changes -> incident resurfaces as active bad
|
|
with previous-generation metadata preserved. Verification passed:
|
|
`go test ./internal/modules/cluster ./internal/platform/runtime ./internal/modules/nodeagent`,
|
|
web-admin `npm run build`, C18Z88 live smoke, and C18Z87 regression smoke.
|
|
Artifact:
|
|
`artifacts/c18z88-service-channel-access-decision-resurface-smoke-result.json`.
|
|
- C18Z89 service-channel access-decision resurface action loop is implemented
|
|
and deployed on docker-test as `rap-backend:fabric-service-channel-0.2.264-c18z89`;
|
|
node-agent remains `rap-node-agent:0.2.256-c18z82` on `test-1/2/3`, and
|
|
web-admin is rebuilt/deployed to `rap_web_admin`. Resurfaced
|
|
access-decision incidents now include `alert_resurfaced_cause`,
|
|
`alert_resurfaced_previous_route_id`, and
|
|
`alert_resurfaced_previous_channel_id`. Web-admin shows the cause beside the
|
|
resurfaced action text. The live smoke proves the operator path:
|
|
access no-safe -> silence current generation -> generation changes and
|
|
resurfaces -> active-channel decision context matches the incident ->
|
|
re-acknowledge current generation -> incident returns to silenced state.
|
|
Verification passed:
|
|
`go test ./internal/modules/cluster ./internal/platform/runtime ./internal/modules/nodeagent`,
|
|
web-admin `npm run build`, C18Z89 live smoke, and C18Z88 regression smoke.
|
|
Artifact:
|
|
`artifacts/c18z89-service-channel-access-decision-resurface-action-smoke-result.json`.
|
|
- C18Z90 service-channel production data-plane contract is implemented and
|
|
deployed on docker-test as `rap-backend:fabric-service-channel-0.2.265-c18z90`;
|
|
node-agent remains `rap-node-agent:0.2.256-c18z82` on `test-1/2/3`, and
|
|
web-admin is rebuilt/deployed to `rap_web_admin`. Service-channel leases now
|
|
include a signed `data_plane` contract in the lease, authority payload,
|
|
introspection response, and lease-maintenance/admin list. The contract
|
|
declares backend API as control-plane transport, fabric service channel over
|
|
fabric routes as working/steady-state data transport, backend relay as
|
|
degraded fallback only, production forwarding required, and service-neutral
|
|
protocol-agnostic logical flow isolation. Web-admin shows data-plane/fallback
|
|
policy in service-channel leases. Verification passed:
|
|
`go test ./internal/modules/cluster ./internal/platform/runtime ./internal/modules/nodeagent`,
|
|
web-admin `npm run build`, C18Z90 live smoke, and C18Z89 regression smoke.
|
|
Artifact:
|
|
`artifacts/c18z90-service-channel-data-plane-contract-smoke-result.json`.
|
|
- C18Z91 node-agent data-plane contract consumption is implemented and
|
|
deployed on docker-test as `rap-node-agent:0.2.266-c18z91` on `test-1/2/3`
|
|
with backend still `rap-backend:fabric-service-channel-0.2.265-c18z90`.
|
|
Service-channel VPN packet ingress now parses signed/introspected
|
|
`data_plane`, validates the production contract, applies the preferred fabric
|
|
route, logs data-plane mode/transports/backend-relay policy/logical-flow
|
|
mode, and reports `data_plane_contract` plus last transport/policy fields in
|
|
heartbeat access telemetry. Verification passed:
|
|
`go test ./cmd/rap-node-agent ./internal/agent ./internal/mesh ./internal/vpnruntime ./internal/config`,
|
|
backend cluster tests, web-admin build, C18Z91 live smoke, and C18Z90
|
|
regression smoke. Artifact:
|
|
`artifacts/c18z91-node-agent-data-plane-contract-enforcement-smoke-result.json`.
|
|
- C18Z92 node-agent backend-fallback policy enforcement is implemented and
|
|
deployed on docker-test as `rap-node-agent:0.2.267-c18z92` on `test-1/2/3`.
|
|
If a signed data-plane contract has `backend_relay_policy=disabled`, the
|
|
service-channel runtime no longer proxies failed/missing fabric-route working
|
|
data through backend relay; it returns a visible service unavailable result.
|
|
The live smoke temporarily disables backend fallback in pool policy, issues a
|
|
no-route lease, verifies `backend_relay_policy=disabled`, posts to test-1,
|
|
and proves the node rejects with 503 instead of backend relay. Verification
|
|
passed: node-agent tests, C18Z92 live smoke, and C18Z91 regression smoke.
|
|
Artifact:
|
|
`artifacts/c18z92-node-agent-disabled-backend-fallback-smoke-result.json`.
|
|
- C18Z93 access-telemetry data-plane projection is implemented and deployed on
|
|
docker-test as `rap-backend:fabric-service-channel-0.2.268-c18z93`;
|
|
node-agent remains `rap-node-agent:0.2.267-c18z92` on `test-1/2/3`, and
|
|
web-admin is rebuilt/deployed to `rap_web_admin`. Backend access telemetry
|
|
now promotes node-reported `data_plane_contract` and last data-plane
|
|
mode/working transport/steady-state transport/backend relay policy/logical
|
|
flow mode to cluster, node, and active-channel diagnostics. Web-admin shows
|
|
summary chips plus channel/node table columns for data-plane adoption and
|
|
relay policy. Verification passed:
|
|
`go test ./internal/modules/cluster ./internal/platform/runtime ./internal/modules/nodeagent`,
|
|
web-admin `npm run build`, C18Z93 live smoke, C18Z92 regression smoke, and
|
|
C18Z91 regression smoke. Artifact:
|
|
`artifacts/c18z93-access-telemetry-data-plane-contract-smoke-result.json`.
|
|
- C18Z94 data-plane contract incident diagnostics are implemented and deployed
|
|
on docker-test as `rap-backend:fabric-service-channel-0.2.269-c18z94`;
|
|
node-agent remains `rap-node-agent:0.2.267-c18z92` on `test-1/2/3`, and
|
|
web-admin is rebuilt/deployed to `rap_web_admin`. Access/rebuild incident
|
|
diagnostics now include `incident_source=data_plane_contract` rows for
|
|
missing data-plane contract reports after accepted traffic, working/steady
|
|
transport mismatches, logical-flow mismatch, disabled backend relay observed,
|
|
and degraded/backend-relay policy violations. The smoke now proves disabled
|
|
backend relay is emitted as a bad incident with action
|
|
`restore_fabric_route_or_change_signed_backend_relay_policy_before_retry`.
|
|
Verification passed:
|
|
`go test ./internal/modules/cluster ./internal/platform/runtime ./internal/modules/nodeagent`,
|
|
web-admin `npm run build`, C18Z94 live smoke, C18Z93 regression smoke, C18Z92
|
|
regression smoke, and C18Z91 regression smoke. Artifact:
|
|
`artifacts/c18z94-data-plane-contract-incident-smoke-result.json`.
|
|
- C18Z95 node-agent blocked-fallback telemetry is implemented and deployed on
|
|
docker-test as backend `rap-backend:fabric-service-channel-0.2.270-c18z95`
|
|
and node-agent `rap-node-agent:0.2.270-c18z95` on `test-1/2/3`; web-admin is
|
|
rebuilt/deployed to `rap_web_admin`. Node-agent now reports
|
|
`backend_fallback_blocked`, `fabric_route_send_failure`, and last data-plane
|
|
violation status/reason in `fabric_service_channel_access_report`. Backend
|
|
access telemetry projects those fields to cluster, node, and active-channel
|
|
rows, and `data_plane_contract` incidents distinguish policy-blocked fallback
|
|
from real backend relay usage. Verification passed: node-agent tests,
|
|
backend tests, web-admin build, C18Z95 live smoke, and C18Z94/C18Z93/C18Z92
|
|
regressions. Artifact:
|
|
`artifacts/c18z95-node-agent-blocked-fallback-telemetry-smoke-result.json`.
|
|
- C18Z96 blocked-fallback rebuild feedback is implemented and deployed on
|
|
docker-test as backend `rap-backend:fabric-service-channel-0.2.281-c18z109`;
|
|
node-agent remains `rap-node-agent:0.2.270-c18z95` on `test-1/2/3`, and
|
|
web-admin remains deployed. Backend now converts heartbeat access reports
|
|
with `fabric_route_send_failed_backend_fallback_blocked` into durable fenced
|
|
`fabric_service_channel_route_feedback` for the active channel primary route.
|
|
The existing route rebuild planner then selects an authorized replacement
|
|
route when one exists. Verification passed: backend tests, node-agent tests,
|
|
web-admin build, C18Z96 live smoke, and C18Z95/C18Z93 regressions. Artifact:
|
|
`artifacts/c18z96-blocked-fallback-rebuild-feedback-smoke-result.json`.
|
|
- C18Z97 blocked-fallback feedback dedup is implemented and deployed on
|
|
docker-test as backend `rap-backend:fabric-service-channel-0.2.281-c18z109`.
|
|
Backend now suppresses repeated access-report-derived route feedback while an
|
|
active fenced/degraded observation from `fabric_service_channel_access_report`
|
|
already exists for the same cluster, reporter node, route, and service class.
|
|
This keeps repeated blocked-fallback send-failure heartbeats from refreshing
|
|
the same feedback and churning rebuild attempts. Verification passed:
|
|
backend tests, node-agent tests, C18Z97 live smoke, and C18Z96/C18Z95
|
|
regressions. Artifact:
|
|
`artifacts/c18z97-blocked-fallback-feedback-dedup-smoke-result.json`.
|
|
- C18Z98 blocked-fallback rebuild correlation is implemented and deployed on
|
|
docker-test as backend `rap-backend:fabric-service-channel-0.2.281-c18z109`;
|
|
web-admin is rebuilt/deployed to `rap_web_admin`. Backend now carries the
|
|
originating access-report route-feedback identity into replacement decisions
|
|
and rebuild-attempt ledger rows: `feedback_observation_id`,
|
|
`feedback_source`, feedback observed/expiry times, channel/resource ids, and
|
|
data-plane violation status/reason. Web-admin shows this correlation in
|
|
Route decisions and Rebuild ledger. Verification passed: backend tests,
|
|
node-agent tests, web-admin build, C18Z98 live smoke, and C18Z97/C18Z96/C18Z95
|
|
regressions. Artifact:
|
|
`artifacts/c18z98-blocked-fallback-rebuild-correlation-smoke-result.json`.
|
|
- C18Z99 rebuild correlation filters are implemented and deployed on
|
|
docker-test as backend `rap-backend:fabric-service-channel-0.2.281-c18z109`;
|
|
web-admin is rebuilt/deployed to `rap_web_admin`. The rebuild-attempt ledger
|
|
API now accepts `feedback_source`, `feedback_channel_id`, and
|
|
`feedback_violation_status` filters, and web-admin exposes them in the
|
|
rebuild ledger filter form. Verification passed: backend tests, node-agent
|
|
tests, web-admin build, C18Z99 live smoke, and C18Z98/C18Z97/C18Z96/C18Z95/
|
|
C18Z93 regressions. Artifact:
|
|
`artifacts/c18z99-rebuild-correlation-filter-smoke-result.json`.
|
|
- C18Z100 rebuild-health feedback breakdown is implemented and deployed on
|
|
docker-test as backend `rap-backend:fabric-service-channel-0.2.281-c18z109`;
|
|
web-admin is rebuilt/deployed to `rap_web_admin`. The rebuild-health summary
|
|
now returns `feedback_breakdowns` grouped by feedback source, feedback
|
|
channel id, and feedback violation status, with total/good/warn/bad/unknown
|
|
counts, active warn/bad counts, silenced count, latest observation time, and
|
|
affected reporter nodes/routes. Web-admin shows the breakdown in the Rebuild
|
|
health panel. Verification passed: backend tests, node-agent tests,
|
|
web-admin build, C18Z100 live smoke, and C18Z99/C18Z98/C18Z97/C18Z96/C18Z95/
|
|
C18Z93 regressions. Artifact:
|
|
`artifacts/c18z100-rebuild-health-feedback-breakdown-smoke-result.json`.
|
|
- C18Z101 rebuild-health feedback drilldown UI is implemented and deployed to
|
|
`rap_web_admin`; backend remains
|
|
`rap-backend:fabric-service-channel-0.2.281-c18z109`. Web-admin now shows
|
|
related incident context on rebuild-health feedback breakdown rows and an
|
|
`open ledger` action that switches to deep rebuild ledger with
|
|
`feedback_source`, `feedback_channel_id`, and `feedback_violation_status`
|
|
prefilled from the selected breakdown. Verification passed: web-admin build
|
|
and deployed asset/download checks.
|
|
- C18Z102 rebuild-health feedback drilldown audit breadcrumbs are implemented
|
|
and deployed on docker-test as backend
|
|
`rap-backend:fabric-service-channel-0.2.281-c18z109`; web-admin is rebuilt/
|
|
deployed to `rap_web_admin`. The existing rebuild investigation endpoint now
|
|
accepts feedback source/channel/violation drilldown payloads and records
|
|
`fabric.service_channel_rebuild_feedback_breakdown.investigation_opened`
|
|
cluster audit events before web-admin opens the filtered deep rebuild ledger.
|
|
Verification passed: backend tests, web-admin build, C18Z102 live smoke, and
|
|
C18Z100/C18Z99/C18Z98 regressions. Artifact:
|
|
`artifacts/c18z102-rebuild-health-feedback-drilldown-audit-smoke-result.json`.
|
|
- C18Z103 Fabric diagnostics drilldown audit visibility is implemented and
|
|
deployed to `rap_web_admin`; backend remains
|
|
`rap-backend:fabric-service-channel-0.2.281-c18z109`. Web-admin now filters
|
|
the loaded cluster audit list for rebuild incident and feedback-breakdown
|
|
investigation events and shows recent drilldowns in the Fabric diagnostics
|
|
panel with time, source, feedback filters, target reporter/route, actor, and
|
|
reason. Verification passed: web-admin build and deployed asset/download
|
|
checks.
|
|
- C18Z104 focused Fabric audit loading is implemented and deployed on
|
|
docker-test as backend `rap-backend:fabric-service-channel-0.2.281-c18z109`;
|
|
web-admin is rebuilt/deployed to `rap_web_admin`. The cluster audit API now
|
|
accepts repeated or comma-separated `event_type` filters plus `target_type`
|
|
filters, and Fabric diagnostics loads recent rebuild incident/feedback
|
|
breakdown investigation breadcrumbs with a dedicated filtered request instead
|
|
of depending on the generic latest-100 audit list. Verification passed:
|
|
backend tests, web-admin build, C18Z104 live smoke, and C18Z102/C18Z100
|
|
regressions. Artifact:
|
|
`artifacts/c18z104-focused-fabric-audit-smoke-result.json`.
|
|
- C18Z105 Fabric drilldown breadcrumb correlation UI is implemented and
|
|
deployed to `rap_web_admin`; backend remains
|
|
`rap-backend:fabric-service-channel-0.2.281-c18z109`. Recent investigation
|
|
rows in Fabric diagnostics now show whether each breadcrumb still matches a
|
|
current rebuild-health feedback breakdown or visible rebuild incident, and
|
|
provide an `open` action to jump back into the matching filtered ledger path.
|
|
Verification passed: web-admin build and deployed asset/download checks.
|
|
- C18Z106 server-side Fabric drilldown breadcrumb correlation is implemented
|
|
and deployed on docker-test as backend
|
|
`rap-backend:fabric-service-channel-0.2.281-c18z109`; web-admin is rebuilt/
|
|
deployed to `rap_web_admin`. Focused audit reads with
|
|
`correlation=fabric_diagnostics` now return `correlation_hints` with current
|
|
diagnostic status and matching rebuild-health feedback breakdown or rebuild
|
|
incident when present. Web-admin consumes those hints and keeps local matching
|
|
as fallback. The rebuild-health feedback breakdown window is raised to 100
|
|
groups after C18Z100 regression exposed the previous cap could hide fresh
|
|
failure classes on noisy test history. Verification passed: backend tests,
|
|
web-admin build, C18Z106 live smoke, and C18Z104/C18Z100 regressions.
|
|
Artifact: `artifacts/c18z106-audit-correlation-hints-smoke-result.json`.
|
|
- C18Z107 drilldown breadcrumb summary is implemented and deployed on
|
|
docker-test as backend `rap-backend:fabric-service-channel-0.2.281-c18z109`;
|
|
web-admin is rebuilt/deployed to `rap_web_admin`. Audit responses now include
|
|
compact `audit_summary` aggregates beside `audit_events`; focused Fabric
|
|
diagnostics uses them to show counts by current diagnostic status, feedback
|
|
source, feedback violation status, correlated/not-visible totals, and latest
|
|
time above the Recent investigations rows. Verification passed: backend
|
|
tests, web-admin build, C18Z107 live smoke, and C18Z106/C18Z104 regressions.
|
|
Artifact: `artifacts/c18z107-audit-correlation-summary-smoke-result.json`.
|
|
- C18Z108 dedicated Fabric diagnostics breadcrumbs are implemented and deployed
|
|
on docker-test as backend `rap-backend:fabric-service-channel-0.2.281-c18z109`;
|
|
web-admin is rebuilt/deployed to `rap_web_admin`. Backend exposes
|
|
`GET /clusters/{clusterID}/fabric/service-channels/rebuild-investigations/breadcrumbs`
|
|
returning `rebuild_investigation_breadcrumbs` with events and summary, so the
|
|
operator Recent investigations workflow no longer overloads the generic
|
|
cluster audit endpoint. Verification passed: backend tests, web-admin build,
|
|
C18Z108 live smoke, and C18Z107/C18Z106/C18Z100 regressions. Artifact:
|
|
`artifacts/c18z108-dedicated-breadcrumbs-smoke-result.json`.
|
|
- C18Z109 Fabric diagnostics breadcrumb freshness windows are implemented and
|
|
deployed on docker-test as backend
|
|
`rap-backend:fabric-service-channel-0.2.281-c18z109`; web-admin is rebuilt/
|
|
deployed to `rap_web_admin`. The dedicated breadcrumb endpoint accepts
|
|
`current_window_seconds` and `history_window_seconds`, annotates events with
|
|
`correlation_hints.breadcrumb_status` (`current`, `stale`, `expired`) plus
|
|
age/window seconds, returns current/stale/expired totals, and includes
|
|
`counts_by_breadcrumb_status` in summary. Web-admin shows freshness chips and
|
|
age in Recent investigations. Verification passed: backend tests, web-admin
|
|
build, C18Z109 live smoke, and C18Z108/C18Z107/C18Z106 regressions. Artifact:
|
|
`artifacts/c18z109-breadcrumb-freshness-window-smoke-result.json`.
|
|
- C19Q Remote Workspace mailbox guardrails are implemented and
|
|
runtime-smoke-proven on docker-test. The adapter-session mailbox handoff now
|
|
has unit and live coverage for invalid adapter session IDs, unknown sessions,
|
|
invalid limits, and bounded `drain=true&limit=N` partial drain semantics.
|
|
This remains probe-only and node-local: it does not enable RDP protocol
|
|
forwarding, desktop frame transport, Android work, or backend relay behavior.
|
|
Verification passed: `go test ./internal/mesh` in `agents/rap-node-agent` and
|
|
`scripts/fabric/c19q-remote-workspace-adapter-mailbox-guardrails-smoke.ps1`.
|
|
Artifact:
|
|
`artifacts/c19q-remote-workspace-adapter-mailbox-guardrails-smoke-result.json`.
|
|
- C19R Remote Workspace mailbox long-poll ergonomics are implemented and
|
|
runtime-smoke-proven on docker-test. The mailbox endpoint now accepts bounded
|
|
`wait_ms`, returns explicit `empty`, `waited`, `wait_timeout`, and `wait_ms`
|
|
fields, and wakes when a delayed mailbox event arrives before timeout.
|
|
Node-agent image `rap-node-agent:codex-service-supervisor-20260512s` is built
|
|
and deployed on `test-1/2/3`. Verification passed:
|
|
`go test ./internal/mesh`, C19R live smoke, and C19Q regression smoke.
|
|
Artifact:
|
|
`artifacts/c19r-remote-workspace-mailbox-long-poll-smoke-result.json`.
|
|
- C19S Remote Workspace mailbox telemetry is implemented and
|
|
runtime-smoke-proven on docker-test. Workload status and heartbeat telemetry
|
|
now expose mailbox read/wait/timeout/empty-read counters plus last mailbox
|
|
read metadata, so adapter consumer polling behavior is visible without
|
|
enabling desktop frame transport. Node-agent image
|
|
`rap-node-agent:codex-service-supervisor-20260512t` is built and deployed on
|
|
`test-1/2/3`. Verification passed: `go test ./internal/mesh`, C19S live
|
|
smoke, and C19R regression smoke. Artifact:
|
|
`artifacts/c19s-remote-workspace-mailbox-telemetry-smoke-result.json`.
|
|
- C19T Remote Workspace mailbox consumer checkpoint/ack metadata is implemented
|
|
and runtime-smoke-proven on docker-test. The mailbox endpoint now accepts a
|
|
validated `consumer_id` and optional `ack_sequence`, returns consumer
|
|
checkpoint/ack/lag/read metadata, and keeps bounded per-session node-local
|
|
consumer cursor state. Workload status and heartbeat telemetry expose
|
|
aggregate/current-session consumer read and ack counters. Node-agent image
|
|
`rap-node-agent:codex-service-supervisor-20260512u` is built and deployed on
|
|
`test-1/2/3`. Verification passed: `go test ./internal/mesh`, C19T live
|
|
smoke, and C19S regression smoke. Artifact:
|
|
`artifacts/c19t-remote-workspace-mailbox-consumer-checkpoint-smoke-result.json`.
|
|
- C19U Remote Workspace mailbox consumer lifecycle guardrails are implemented
|
|
and runtime-smoke-proven on docker-test. Consumers can pass
|
|
`reset_consumer=true` with a validated `consumer_id` to clear cursor state
|
|
before the current read is recorded. Mailbox responses expose consumer
|
|
count/capacity, created/reset/evicted lifecycle flags, and consumer
|
|
timestamps; workload status and heartbeat telemetry expose consumer reset and
|
|
eviction counters. Node-agent image
|
|
`rap-node-agent:codex-service-supervisor-20260512v` is built and deployed on
|
|
`test-1/2/3`. Verification passed: `go test ./internal/mesh`, C19U live
|
|
smoke, and C19T regression smoke. Artifact:
|
|
`artifacts/c19u-remote-workspace-mailbox-consumer-lifecycle-smoke-result.json`.
|
|
- C19V Remote Workspace mailbox consumer cursor inspection is implemented and
|
|
runtime-smoke-proven on docker-test. Active adapter sessions now expose a
|
|
read-only
|
|
`/mesh/v1/remote-workspace/adapter-sessions/{adapter_session_id}/mailbox/consumers`
|
|
endpoint with bounded cursor snapshots: consumer ids, checkpoint/ack
|
|
sequences, lag, read/ack totals, and timestamps. The endpoint is read-only and
|
|
does not increment mailbox reads, acks, resets, or drain events. Node-agent
|
|
image `rap-node-agent:codex-service-supervisor-20260512w` is built and
|
|
deployed on `test-1/2/3`. Verification passed: `go test ./internal/mesh`,
|
|
C19V live smoke, and C19U regression smoke. Artifact:
|
|
`artifacts/c19v-remote-workspace-mailbox-consumer-snapshot-smoke-result.json`.
|
|
- C19W Remote Workspace mailbox cursor-aware resume reads are implemented and
|
|
runtime-smoke-proven on docker-test. The mailbox endpoint now accepts
|
|
`after_sequence` for non-destructive reads, returns `skipped_count` and
|
|
`returned_count`, and long-polls for events newer than the requested sequence.
|
|
`after_sequence` with `drain=true` is rejected to keep resume reads separate
|
|
from destructive drains. Node-agent image
|
|
`rap-node-agent:codex-service-supervisor-20260512x` is built and deployed on
|
|
`test-1/2/3`. Verification passed: `go test ./internal/mesh`, C19W live
|
|
smoke, and C19V regression smoke. Artifact:
|
|
`artifacts/c19w-remote-workspace-mailbox-after-sequence-smoke-result.json`.
|
|
- C19X Remote Workspace mailbox consumer-aware resume is implemented and
|
|
runtime-smoke-proven on docker-test. Mailbox reads with `consumer_id` can pass
|
|
`resume_from=ack|checkpoint`; the node-agent resolves the stored cursor to
|
|
`after_sequence` before reading and returns `resume_from`/`resume_sequence`.
|
|
Guardrails reject mixing resume with manual `after_sequence`, drain, reset,
|
|
missing consumers, or invalid cursor names. Node-agent image
|
|
`rap-node-agent:codex-service-supervisor-20260512y` is built and deployed on
|
|
`test-1/2/3`. Verification passed: `go test ./internal/mesh`, C19X live
|
|
smoke, and C19W regression smoke. Artifact:
|
|
`artifacts/c19x-remote-workspace-mailbox-consumer-resume-smoke-result.json`.
|
|
- C19Y Remote Workspace mailbox resume telemetry is implemented and
|
|
runtime-smoke-proven on docker-test. Workload status and heartbeat telemetry
|
|
now expose resume/after-sequence read totals, returned/skipped totals, and the
|
|
last resume cursor/sequence/consumer plus returned/skipped counts for
|
|
operator diagnostics. Session snapshots include the same per-session resume
|
|
counters. Node-agent image
|
|
`rap-node-agent:codex-service-supervisor-20260512z` is built and deployed on
|
|
`test-1/2/3`. Verification passed: `go test ./internal/mesh`, C19Y live
|
|
smoke, C19X source smoke, and C19W regression smoke. Artifact:
|
|
`artifacts/c19y-remote-workspace-mailbox-resume-telemetry-smoke-result.json`.
|
|
- C19Z Remote Workspace adapter runtime readiness summary is implemented and
|
|
runtime-smoke-proven on docker-test. The sink report now includes compact
|
|
`adapter_runtime_readiness` diagnostics with session lifecycle state, mailbox
|
|
depth, consumer cursor, resume cursor, skipped/returned counts, and
|
|
ready/diagnostic status for operator handoff checks. Node-agent image
|
|
`rap-node-agent:codex-service-supervisor-20260512z1` is built and deployed on
|
|
`test-1/2/3`. Verification passed: `go test ./internal/mesh`, C19Z live
|
|
smoke, C19X source smoke, and C19Y regression smoke. Artifact:
|
|
`artifacts/c19z-remote-workspace-adapter-readiness-smoke-result.json`.
|
|
- C19Z1 Remote Workspace mailbox handoff preflight is implemented and
|
|
runtime-smoke-proven on docker-test. The node-agent now exposes read-only
|
|
`GET /mesh/v1/remote-workspace/adapter-sessions/{adapter_session_id}/mailbox/preflight`
|
|
for `consumer_id` plus `resume_from=ack|checkpoint`; it validates the cursor
|
|
and reports the expected next event window without reading, draining, acking,
|
|
or mutating consumer state. Node-agent image
|
|
`rap-node-agent:codex-service-supervisor-20260512z2` is built and deployed on
|
|
`test-1/2/3`. Verification passed: `go test ./internal/mesh`, C19Z1 live
|
|
smoke, C19X source smoke, and C19Z regression smoke. Artifact:
|
|
`artifacts/c19z1-remote-workspace-mailbox-preflight-smoke-result.json`.
|
|
|
|
The current phase is NOT:
|
|
- full mesh routing implementation
|
|
- full VPN orchestration
|
|
- multi-cluster runtime traffic handling
|
|
- production data-plane migration
|
|
- complete updater rollout orchestration
|
|
- video meetings
|
|
- final native client UI redesign
|
|
|
|
Future mesh, VPN, multi-cluster, node-agent updater, and production realtime data-plane work must be introduced only through explicit, narrow, staged implementation prompts.
|
|
|
|
Always keep the project production-oriented. Do not simplify it into a toy app.
|