2079 lines
84 KiB
Markdown
2079 lines
84 KiB
Markdown
# Cluster, Node, Mesh, and Admin Foundation
|
|
|
|
Status: target foundation plan plus staged implementation tracker. This document defines the next platform-core direction and implementation order. It is not an instruction to rewrite the current RDP MVP.
|
|
|
|
The current RDP work is paused by product decision. The proven RDP access core remains valuable and must not be broken. The next platform step is to build the cluster, node enrollment, mesh preparation, and platform administration foundation that the Secure Access Fabric will depend on.
|
|
|
|
Related documents:
|
|
|
|
- `docs/architecture/SECURE_ACCESS_FABRIC_TARGET.md`
|
|
- `docs/architecture/DATA_PLANE_V1.md`
|
|
- `docs/architecture/WEB_INGRESS_AND_ADMIN_UI_MODEL.md`
|
|
- `docs/codex/CURRENT_STATUS.md`
|
|
|
|
## Purpose
|
|
|
|
The platform is a Secure Access Fabric, not only an RDP proxy. RDP is the first managed service adapter, but the long-term platform must support additional adapters and network services such as VNC, SSH, VPN/IP tunnel, internal web apps, file services, video/audio, update cache, and relay/entry nodes.
|
|
|
|
The lower platform foundation is the RAP Fabric Core: a distributed runtime
|
|
layer above the host OS and below service adapters. It is not a real operating
|
|
system. It is implemented through native `rap-node-agent`, control-plane
|
|
contracts, signed scoped cluster snapshots, node-local state, and service
|
|
supervision boundaries.
|
|
|
|
Before implementing mesh traffic, VPN runtime, or organization administration UI, the platform needs a clear foundation for:
|
|
|
|
- clusters
|
|
- host-level node identity
|
|
- node enrollment
|
|
- Fabric Core terminology and contracts
|
|
- scoped configuration distribution
|
|
- node-local state
|
|
- role assignment
|
|
- service workload placement
|
|
- platform-owner administration
|
|
- future organization administration
|
|
- safe multi-cluster operations
|
|
|
|
## Current Foundation Inventory
|
|
|
|
The backend already contains an early platform-core foundation:
|
|
|
|
- `nodes`
|
|
- `node_capabilities`
|
|
- `node_services`
|
|
- `node_update_policies`
|
|
- `node_partition_states`
|
|
- `node_agent_update_runs`
|
|
- node management routes under `/nodes`
|
|
- node-agent routes under `/node-agents`
|
|
|
|
This is useful, but it is not sufficient as the final node and cluster model. Current gaps:
|
|
|
|
- no explicit first-class `clusters`
|
|
- no explicit cluster membership model
|
|
- no join request approval workflow
|
|
- no short-lived join token model
|
|
- no node certificate lifecycle model
|
|
- no clear separation between technical capability and authorized role assignment
|
|
- no platform admin console model
|
|
- no multi-cluster administration model
|
|
- no organization-safe visibility model for entry/egress/service endpoints
|
|
- node-agent registration is too direct for production trust boundaries
|
|
|
|
## Vocabulary
|
|
|
|
Platform:
|
|
The whole Secure Access Fabric installation operated by the platform owner.
|
|
|
|
Cluster:
|
|
A bounded control and trust domain containing nodes, policies, service placements, certificates, health state, and routing state. A platform may operate multiple clusters.
|
|
|
|
Organization:
|
|
A tenant. Organizations own or are granted access to resources, policies, sessions, services, and selected entry/egress points. Organizations must not see full internal mesh topology.
|
|
|
|
Fabric Core:
|
|
The lower OS-like distributed runtime layer of the Secure Access Fabric. It runs
|
|
above the host OS and is owned by native `rap-node-agent` plus control-plane
|
|
contracts. It owns node identity, enrollment, local state, capability reporting,
|
|
role assignment consumption, signed scoped configuration snapshots, update
|
|
trust, and service supervision boundaries.
|
|
|
|
Node:
|
|
A host-level identity. A node is not a Docker container. A node is managed by native `rap-node-agent`.
|
|
|
|
Node Agent:
|
|
Native host process responsible for node identity, enrollment, certificates, health, role execution, service workload supervision, updates, recovery, and reporting.
|
|
|
|
Service Workload:
|
|
A workload executed on a node. It may be native or containerized. Examples: `rdp-worker`, `vnc-worker`, `entry-node`, `relay-node`, `file-storage-cache`.
|
|
|
|
Public/Admin HTTPS Ingress:
|
|
A service-edge role that listens on TCP `80`/`443` for browser/API HTTPS and
|
|
forwards accepted requests into the QUIC-only fabric service channel. It is not
|
|
an authority service and does not imply permission to manage the cluster.
|
|
|
|
Admin UI Runtime:
|
|
A scoped admin service runtime. Global admin runtime may run only on
|
|
platform-owner trusted nodes; cluster, organization, and user portal runtimes
|
|
receive only their scoped projections.
|
|
|
|
Capability:
|
|
What a node can technically do. Example: `can_run_rdp_worker`.
|
|
|
|
Role Assignment:
|
|
What a node is authorized to do in a cluster and, where relevant, for an organization. Capabilities are not permissions.
|
|
|
|
Join Request:
|
|
A request from a node-agent to join a cluster, usually created with a short-lived join token and approved by a platform admin.
|
|
|
|
Entry Node:
|
|
Accepts client connections and maps them into platform logical channels.
|
|
|
|
Core Mesh Node:
|
|
Maintains secure overlay routing and QoS. It should remain protocol-neutral.
|
|
|
|
Egress / Service Node:
|
|
Hosts service adapters or network exits that connect to target resources.
|
|
|
|
Service Adapter:
|
|
Protocol-specific edge function, for example RDP Adapter, VNC Adapter, SSH Adapter, HTTP/internal app adapter, or video/audio adapter.
|
|
|
|
Remote Server/Desktop Access:
|
|
Product-level managed access service consumed by Access Client. RDP, VNC, and
|
|
SSH are internal protocol adapters selected from the organization resource
|
|
protocol; they are not separate organization-facing cluster services.
|
|
|
|
Egress Pool:
|
|
Organization-visible logical exit such as "Office Moscow" or "Datacenter
|
|
Kazan". It may be backed by multiple internal nodes. Organizations reference
|
|
egress pools, not concrete nodes or mesh topology.
|
|
|
|
Fabric Routing Engine:
|
|
Logical fabric-layer component responsible for peer selection, route discovery,
|
|
route scoring, failover, shortcut decisions, route cache management, topology
|
|
hiding, and channel-aware routing. It is not a runtime implementation in the
|
|
current phase.
|
|
|
|
Fabric Storage Service / Config Storage Service:
|
|
Future scoped storage/distribution service for signed cluster snapshots, peer
|
|
directories, policy snapshots, update metadata, and node-scoped configuration.
|
|
PostgreSQL remains authoritative.
|
|
|
|
Version Storage / Update Repository:
|
|
Logical artifact repository for signed platform, node-agent, and service
|
|
workload releases. It stores release manifests, OS/architecture artifacts,
|
|
stable/current/candidate channel metadata, hashes, signatures, compatibility
|
|
metadata, and migration bundles. PostgreSQL remains authoritative for rollout
|
|
policy, approvals, and audit.
|
|
|
|
## Golden Rules
|
|
|
|
1. Node identity belongs to the native `rap-node-agent`, not to a container.
|
|
2. Containers are packaging and isolation boundaries only.
|
|
3. Capabilities are not permissions.
|
|
4. Role assignment must be explicit per cluster and, when needed, per organization.
|
|
5. Platform admins operate cluster/node trust and placement.
|
|
6. Organization admins manage only organization-visible resources, policies, sessions, and allowed service endpoints.
|
|
7. Organizations must not see intermediate core mesh topology.
|
|
8. Mesh routing must not be implemented before enrollment, identity, and role assignment are trustworthy.
|
|
9. Disconnected minority partitions must not add nodes or change policy.
|
|
10. PostgreSQL remains the source of truth; Redis/live state must be reconstructable.
|
|
11. RDP MVP behavior must remain preserved while platform-core work progresses.
|
|
12. Fabric Core comes before mesh runtime.
|
|
13. Service adapters must not own mesh routing logic.
|
|
14. Nodes must not store or learn more cluster, organization, secret, or route data than their role requires.
|
|
15. Config/storage services distribute scoped snapshots and caches; they are not a second source of truth.
|
|
16. Redis must not store durable topology, durable configuration, node identity, policy, or organization data.
|
|
17. Fabric routing decisions must not depend on live backend availability.
|
|
18. Fabric Storage / Config Storage must not become a general-purpose distributed database.
|
|
19. Version Storage stores signed artifacts and release manifests; it does not
|
|
decide rollout policy and must not serve unsigned or unapproved artifacts.
|
|
20. Node-agent is the local supervisor for health, restart, update, and rollback
|
|
of node services, but Control Plane owns rollout policy and durable schema
|
|
migration orchestration.
|
|
21. HTTP/HTTPS is an external service edge only. Fabric node-to-node transport
|
|
remains QUIC-only.
|
|
22. A node that accepts `443` does not own management authority. Admin authority
|
|
belongs to signed roles, scoped claims, policy, and trusted runtime nodes.
|
|
23. Global admin runtime, policy authority, and audit sink must run only on
|
|
platform-owner controlled nodes. Organization and cluster portals must not
|
|
expose unrelated tenants, clusters, or internal mesh topology.
|
|
|
|
## Existing Node Management Semantics
|
|
|
|
The Platform Owner Console must treat a node as a concrete host-level identity,
|
|
not as an abstract request for any free capacity. A cluster membership always
|
|
connects a specific `node_id` to a specific `cluster_id`.
|
|
|
|
The same physical node may participate in multiple clusters only through
|
|
isolated cluster memberships, certificates, tokens, storage namespaces, and
|
|
policies. Role assignments and desired service workloads are cluster-scoped.
|
|
This means a node may run `entry-node` for one cluster, have no ingress role in
|
|
another cluster, and still report the same underlying host capabilities.
|
|
|
|
Node capabilities are reported by `rap-node-agent` as the technical upper bound
|
|
of what the host can do. Capabilities are not permissions. The Control Plane
|
|
must expose to each cluster only the roles and service workloads that are
|
|
explicitly assigned for that cluster and, where relevant, for a specific
|
|
organization.
|
|
|
|
Current owner-console operations for existing nodes are:
|
|
|
|
- view active-cluster nodes
|
|
- view all physical nodes visible to the platform owner
|
|
- inspect cluster membership state per node
|
|
- organize active-cluster nodes into hierarchical node groups
|
|
- inspect heartbeat, telemetry, reported services, and desired services
|
|
- assign cluster-scoped roles
|
|
- enable or disable cluster-scoped desired service workloads
|
|
- manage node participation in product-level functions such as Remote
|
|
Server/Desktop Access, without exposing protocol adapters as separate
|
|
organization-facing services
|
|
- enable node/cluster/organization test telemetry flags
|
|
- disable a node membership in the active cluster
|
|
- revoke a node identity as a high-risk platform-owner action
|
|
|
|
Creating a brand-new node is not a direct database insert. The future production
|
|
flow remains installation of native `rap-node-agent` on the host, enrollment
|
|
with a short-lived join token, platform-owner approval, then explicit role and
|
|
service assignment. This enrollment/create-node UX is intentionally separate
|
|
from existing-node management.
|
|
|
|
Node groups are cluster-scoped inventory folders, not node identities and not
|
|
permissions. A group may contain child groups, and a node membership may be
|
|
assigned to one group in that cluster. The same physical node may be in a
|
|
different group, or no group, in another cluster. Groups are intended for
|
|
large-scale operator usability: data centers, sites, customers, regions,
|
|
environments, ownership boundaries, or maintenance rings. Authorization still
|
|
comes from cluster membership, role assignment, organization policy, and
|
|
service desired state, not from the visual group tree.
|
|
|
|
Remote Server/Desktop Access is managed as one product service. A node may be
|
|
allowed to run this access service, while the concrete protocol adapter is chosen
|
|
from the organization resource (`rdp`, `vnc`, `ssh`). Cluster admins should not
|
|
configure separate organization-facing RDP/VNC/SSH services; those adapters are
|
|
internal runtime implementations.
|
|
|
|
## Fabric Core Foundation
|
|
|
|
Layer order:
|
|
|
|
1. Host OS
|
|
2. RAP Fabric Core
|
|
3. Secure Fabric Network
|
|
4. Service Runtime / Service Adapters
|
|
5. Access Clients / Admin UI
|
|
|
|
Fabric Core responsibilities:
|
|
|
|
- node identity
|
|
- node enrollment
|
|
- local node state
|
|
- capability reporting
|
|
- role assignment consumption
|
|
- signed scoped cluster snapshots
|
|
- update trust
|
|
- service workload supervision boundary
|
|
|
|
Fabric Core does not implement service protocols. RDP, VNC, SSH, VPN, video,
|
|
and file transfer are services above the fabric. They consume node identity,
|
|
role assignment, local state, routing, and policy from the Fabric Core.
|
|
|
|
Fabric Core must be trustworthy before mesh runtime traffic exists. Mesh
|
|
routing, VPN/IP tunnel runtime, relay packet routing, and service workload
|
|
execution must not be implemented before node identity, enrollment, trust,
|
|
role assignment, and scoped configuration distribution are solid.
|
|
|
|
## Peer Discovery and Routing Foundation
|
|
|
|
Nodes must not maintain active connections to all other nodes.
|
|
|
|
Each node maintains:
|
|
|
|
- active peers
|
|
- warm candidate peers
|
|
- cold / bootstrap peers
|
|
|
|
Recommended defaults:
|
|
|
|
- normal node: 3-5 active peers
|
|
- relay / core node: 8-20 active peers
|
|
- thin / mobile node: 1-3 active peers
|
|
|
|
Peer selection must be score-based, not latency-only.
|
|
|
|
Score inputs:
|
|
|
|
- latency
|
|
- packet loss
|
|
- reliability
|
|
- region distance
|
|
- node load
|
|
- bandwidth availability
|
|
- role suitability
|
|
- policy constraints
|
|
- trust level
|
|
- recent failure history
|
|
|
|
Service adapters request a destination node, resource target, egress node, or
|
|
egress pool. The Fabric Routing Engine chooses the path. RDP/VNC/SSH/VPN/video
|
|
adapters must not implement topology discovery, multi-hop route selection,
|
|
shortcut creation, or cross-cluster routing policy.
|
|
|
|
Fabric routing uses node-local state, signed scoped snapshots, peer cache, and
|
|
route cache. It must respect policy, organization scope, cluster boundaries,
|
|
and partition/authority state. Service adapters must not select routes,
|
|
discover peers, manage mesh connections, implement failover, implement shortcut
|
|
logic, implement partition recovery, or implement cross-cluster routing policy.
|
|
|
|
## Node-Local State and Scoped Configuration
|
|
|
|
`rap-node-agent` local state should include:
|
|
|
|
- node identity
|
|
- cluster membership
|
|
- signed scoped cluster snapshot
|
|
- peer cache
|
|
- route cache
|
|
- service assignment cache
|
|
- local health state
|
|
- partition / degraded state
|
|
- last applied config version
|
|
- pending update metadata
|
|
|
|
A node must not store full cluster topology unless its role requires it.
|
|
|
|
Configuration distribution is need-to-know:
|
|
|
|
- core mesh nodes receive neighbor/peer data, route policy, QoS policy, and cluster version
|
|
- ingress nodes receive allowed client entry policies, token validation config, and route entry data
|
|
- egress/service nodes receive assigned service configs, needed resource refs, connector refs, and service policy
|
|
- storage services receive assigned shard/scope data and replication metadata
|
|
|
|
Secrets are delivered only through approved resolvers and only when required at
|
|
runtime. RDP credentials, organization user lists, unrelated storage shards, and
|
|
full topology must not be distributed to nodes that do not need them.
|
|
|
|
## Fabric Storage / Config Storage Foundation
|
|
|
|
Fabric Storage Service / Config Storage Service is a future foundation service,
|
|
not a replacement for PostgreSQL.
|
|
|
|
Purpose:
|
|
|
|
- store and replicate scoped cluster configuration
|
|
- distribute signed snapshots
|
|
- keep frequently used data near services
|
|
- support local high-speed reads
|
|
- preserve configuration availability when some nodes disappear
|
|
|
|
Rules:
|
|
|
|
- not every node stores all data
|
|
- replication factor is policy-driven
|
|
- critical data is replicated across failure domains
|
|
- hot data is placed near services that use it
|
|
- organization and cluster isolation are mandatory
|
|
- PostgreSQL remains authoritative
|
|
- storage/config services are distribution/cache layers only
|
|
- storage/config services must not accept direct node writes as authoritative state
|
|
- storage/config services must not expose arbitrary query capabilities
|
|
- storage/config services must not store full cluster or organization data on every node
|
|
|
|
Data classes:
|
|
|
|
- platform global data
|
|
- cluster state
|
|
- organization data
|
|
- service config data
|
|
- realtime / lease state
|
|
- audit / event data
|
|
- artifacts / update data
|
|
|
|
## Version Storage and Node Update Foundation
|
|
|
|
The platform includes a logical Version Storage / Update Repository service.
|
|
|
|
It stores approved release artifacts and metadata for:
|
|
|
|
- native `rap-node-agent`
|
|
- service workloads such as `rdp-worker`, `entry-node`, `relay-node`,
|
|
`file-storage-cache`, `update-cache`, future VNC/SSH/VPN services
|
|
- OS / architecture-specific builds
|
|
- migration bundles for data structure changes
|
|
|
|
Release channels:
|
|
|
|
- `stable`: known-good version used for rollback
|
|
- `current`: normal intended version for the cluster/ring
|
|
- `candidate`: version staged for test/canary rollout
|
|
|
|
Rules:
|
|
|
|
- PostgreSQL is authoritative for rollout policy, channel assignment, approvals,
|
|
target nodes/rings, and audit.
|
|
- Version Storage stores immutable signed artifacts and manifests.
|
|
- `update-cache` mirrors approved artifacts close to nodes but does not approve
|
|
or invent versions.
|
|
- Node-agent downloads from authorized update-cache or Version Storage, verifies
|
|
signatures and hashes, stages updates, runs health checks, and reports state.
|
|
- Node-agent may restart workloads and roll back to last stable / last known
|
|
good when health checks fail.
|
|
- Node-agent self-update requires stronger safeguards than normal service
|
|
workloads: A/B slots or equivalent staged replacement, watchdog/recovery, and
|
|
rollback that survives process crash.
|
|
|
|
Data structure migrations:
|
|
|
|
- code updates and data structure updates are separate concerns in one signed
|
|
release manifest
|
|
- migration bundles must declare source version, target version, compatibility,
|
|
preflight checks, backup/snapshot requirements, rollback behavior, and whether
|
|
downgrade is possible
|
|
- Control Plane/PostgreSQL schema migrations are orchestrated by the Control
|
|
Plane release process
|
|
- service-local, cache-local, and node-local state migrations may be executed by
|
|
node-agent only when included in an approved signed manifest
|
|
- failed migrations must leave the node rolled back, degraded, or fenced, never
|
|
silently half-updated
|
|
|
|
Suggested update states:
|
|
|
|
- `idle`
|
|
- `checking`
|
|
- `downloading`
|
|
- `verifying`
|
|
- `staging`
|
|
- `migrating`
|
|
- `starting`
|
|
- `health_checking`
|
|
- `promoting`
|
|
- `running`
|
|
- `rollback_required`
|
|
- `rolling_back`
|
|
- `rolled_back`
|
|
- `failed`
|
|
- `fenced`
|
|
|
|
## Target Data Model
|
|
|
|
The following entities should become the explicit foundation. Names are target names and may be adjusted during implementation if the existing schema requires compatibility.
|
|
|
|
### Cluster
|
|
|
|
`clusters`
|
|
|
|
Purpose:
|
|
First-class cluster records.
|
|
|
|
Key fields:
|
|
|
|
- `id`
|
|
- `name`
|
|
- `slug`
|
|
- `status`
|
|
- `region`
|
|
- `metadata`
|
|
- `created_at`
|
|
- `updated_at`
|
|
|
|
### Cluster Membership
|
|
|
|
`cluster_memberships`
|
|
|
|
Purpose:
|
|
Links nodes to clusters with membership state.
|
|
|
|
Key fields:
|
|
|
|
- `cluster_id`
|
|
- `node_id`
|
|
- `membership_status`
|
|
- `joined_at`
|
|
- `last_seen_at`
|
|
- `metadata`
|
|
|
|
### Node
|
|
|
|
`nodes`
|
|
|
|
Purpose:
|
|
Host-level identity record.
|
|
|
|
Existing `nodes` should be preserved and migrated safely. A default cluster backfill is acceptable for existing rows.
|
|
|
|
Target additions or refinements:
|
|
|
|
- explicit cluster membership through `cluster_memberships`
|
|
- enrollment status separated from runtime health
|
|
- identity/certificate state
|
|
- ownership and organization association only where applicable
|
|
|
|
### Node Identity
|
|
|
|
`node_identities`
|
|
|
|
Purpose:
|
|
Stores node public identity material and trust state.
|
|
|
|
Key fields:
|
|
|
|
- `node_id`
|
|
- `public_key`
|
|
- `certificate_serial`
|
|
- `certificate_not_before`
|
|
- `certificate_not_after`
|
|
- `identity_status`
|
|
- `rotated_at`
|
|
- `revoked_at`
|
|
|
|
Private keys must never be stored in the control plane.
|
|
|
|
### Node Join Token
|
|
|
|
`node_join_tokens`
|
|
|
|
Purpose:
|
|
Short-lived token created by platform admin to allow enrollment into a cluster.
|
|
|
|
Key fields:
|
|
|
|
- `id`
|
|
- `cluster_id`
|
|
- `token_hash`
|
|
- `scope`
|
|
- `expires_at`
|
|
- `max_uses`
|
|
- `used_count`
|
|
- `created_by_user_id`
|
|
- `revoked_at`
|
|
|
|
Raw join tokens must not be stored.
|
|
|
|
### Node Join Request
|
|
|
|
`node_join_requests`
|
|
|
|
Purpose:
|
|
Approval workflow for node enrollment.
|
|
|
|
Key fields:
|
|
|
|
- `id`
|
|
- `cluster_id`
|
|
- `node_name`
|
|
- `node_fingerprint`
|
|
- `public_key`
|
|
- `reported_capabilities`
|
|
- `reported_facts`
|
|
- `requested_roles`
|
|
- `status`
|
|
- `reviewed_by_user_id`
|
|
- `reviewed_at`
|
|
- `approved_node_id`
|
|
- `rejection_reason`
|
|
|
|
Default behavior should be manual approval.
|
|
|
|
### Node Capability
|
|
|
|
`node_capabilities`
|
|
|
|
Purpose:
|
|
Technical capabilities reported by node-agent and verified by platform policy.
|
|
|
|
Capability examples:
|
|
|
|
- `can_accept_client_ingress`
|
|
- `can_accept_node_ingress`
|
|
- `can_route_mesh`
|
|
- `can_egress_internet`
|
|
- `can_egress_private_network`
|
|
- `can_run_rdp_worker`
|
|
- `can_run_vnc_worker`
|
|
- `can_run_vpn_exit`
|
|
- `can_run_vpn_connector`
|
|
- `can_run_file_cache`
|
|
- `can_run_update_cache`
|
|
- `can_run_video_relay`
|
|
|
|
### Node Role Assignment
|
|
|
|
`node_role_assignments`
|
|
|
|
Purpose:
|
|
Explicit authorization for a node to execute roles in a cluster and optionally for specific organizations.
|
|
|
|
Key fields:
|
|
|
|
- `cluster_id`
|
|
- `node_id`
|
|
- `role`
|
|
- `organization_id`
|
|
- `status`
|
|
- `assigned_by_user_id`
|
|
- `assigned_at`
|
|
- `policy`
|
|
|
|
Examples:
|
|
|
|
- `entry-node`
|
|
- `relay-node`
|
|
- `rdp-worker`
|
|
- `vnc-worker`
|
|
- `vpn-exit`
|
|
- `vpn-connector`
|
|
- `file-storage-cache`
|
|
- `update-cache`
|
|
- `video-relay`
|
|
|
|
### Node Service Instance
|
|
|
|
`node_service_instances`
|
|
|
|
Purpose:
|
|
Tracks running service workloads supervised by node-agent.
|
|
|
|
Key fields:
|
|
|
|
- `cluster_id`
|
|
- `node_id`
|
|
- `service_type`
|
|
- `service_instance_id`
|
|
- `desired_state`
|
|
- `reported_state`
|
|
- `version`
|
|
- `last_heartbeat_at`
|
|
- `metadata`
|
|
|
|
### Node Heartbeat
|
|
|
|
`node_heartbeats`
|
|
|
|
Purpose:
|
|
Durable heartbeat observations and latest state snapshots.
|
|
|
|
This can be split into an append-only heartbeat log and a compact latest-state table if needed.
|
|
|
|
### Cluster Audit Event
|
|
|
|
`cluster_audit_events`
|
|
|
|
Purpose:
|
|
Audit cluster, node, enrollment, role, trust, and service-placement changes.
|
|
|
|
Events should include:
|
|
|
|
- join token created/revoked
|
|
- join request received/approved/rejected
|
|
- node certificate issued/rotated/revoked
|
|
- role assigned/removed
|
|
- service desired state changed
|
|
- node health changed
|
|
- cluster partition detected/resolved
|
|
|
|
### Future Mesh Entities
|
|
|
|
These are not Stage C1 implementation requirements, but the data model should not block them:
|
|
|
|
- `mesh_links`
|
|
- `mesh_routes`
|
|
- `cluster_route_intents`
|
|
- `cluster_partitions`
|
|
- `cluster_authority_terms`
|
|
|
|
## Node Enrollment Flow
|
|
|
|
Default enrollment must be explicit and reviewable.
|
|
|
|
1. Platform admin chooses or creates a cluster.
|
|
2. Platform admin creates a short-lived join token scoped to that cluster.
|
|
3. Operator installs native `rap-node-agent` on the host.
|
|
4. Node-agent generates a local keypair.
|
|
5. Node-agent sends a join request with token, public key, node facts, and capabilities.
|
|
6. Control plane validates token hash, expiry, scope, and usage limit.
|
|
7. Control plane creates `node_join_request` in `pending` state.
|
|
8. Platform admin reviews node facts and approves or rejects the request.
|
|
9. On approval, control plane creates or activates the node identity.
|
|
10. Control plane issues cluster-scoped node certificate or certificate metadata.
|
|
11. Node-agent stores identity material locally.
|
|
12. Node-agent starts heartbeats.
|
|
13. Platform admin assigns roles.
|
|
14. Node-agent starts only authorized service workloads.
|
|
|
|
Auto-approval may exist later only for tightly scoped platform-managed bootstrap tokens. It must not be the default.
|
|
|
|
## Node Runtime Expectations
|
|
|
|
`rap-node-agent` should run natively on the host.
|
|
|
|
The agent owns:
|
|
|
|
- host identity
|
|
- node certificates
|
|
- enrollment
|
|
- heartbeat
|
|
- desired role polling
|
|
- service workload supervision
|
|
- update trust and rollback
|
|
- local recovery
|
|
|
|
Containerized workloads are preferred for:
|
|
|
|
- `rdp-worker`
|
|
- `vnc-worker`
|
|
- `relay-node`
|
|
- `entry-node`
|
|
- `file-storage-cache`
|
|
- `update-cache`
|
|
- `video-relay`
|
|
|
|
Native workloads are preferred for:
|
|
|
|
- `vpn-exit`
|
|
- `vpn-connector`
|
|
- host route manager
|
|
- firewall/QoS manager
|
|
- Windows virtual adapter service
|
|
- Android VpnService client
|
|
|
|
Realtime container workloads should avoid Docker bridge/NAT hot paths where low latency matters. Prefer host networking or native mode for latency-critical paths.
|
|
|
|
Privileged containers are discouraged. If a workload needs `NET_ADMIN`, `/dev/net/tun`, host firewall control, or broad host privileges, native mode should be preferred unless explicitly approved.
|
|
|
|
## Admin Console Model
|
|
|
|
Admin UI is served through Web Ingress, but cluster configuration belongs to
|
|
the Control Plane.
|
|
|
|
The Web/Admin boundary is defined in
|
|
`docs/architecture/WEB_INGRESS_AND_ADMIN_UI_MODEL.md`.
|
|
|
|
Rules:
|
|
|
|
- Web Ingress is an HTTP/HTTPS entry and presentation layer only
|
|
- Web Ingress must not own cluster state, node trust, policy, or secrets
|
|
- Admin pages may be dynamically composed from safe `ui_manifest` / page
|
|
definitions
|
|
- dynamic page definitions must be schema-driven and must not contain internal
|
|
topology, peer caches, route caches, secrets, raw credentials, or arbitrary
|
|
executable code
|
|
- every cluster mutation still goes through Control Plane authorization,
|
|
PostgreSQL source-of-truth mutation, and audit
|
|
- Fabric Storage / Config Storage role does not imply admin/web ingress role
|
|
- adding a storage node to a cluster must not move or create the cluster panel
|
|
automatically
|
|
|
|
### Admin Endpoint Placement
|
|
|
|
Admin UI has three distinct scopes:
|
|
|
|
- Platform Owner Console: global platform-owner scope. It may aggregate
|
|
visibility across clusters according to platform policy and audit.
|
|
- Cluster Admin Endpoint: cluster-local admin/web ingress service for one
|
|
cluster. It is served only by nodes explicitly assigned an approved
|
|
admin/web ingress role.
|
|
- Organization Admin Panel: tenant-safe projection over allowed organization
|
|
resources, endpoints, policies, sessions, and safe status.
|
|
|
|
Storage nodes are configuration distribution/cache nodes. They are not browser
|
|
admin entry points and they do not own platform or cluster administration.
|
|
|
|
A cluster-local admin endpoint may become available only after:
|
|
|
|
- the cluster has healthy authority state
|
|
- scoped configuration snapshots are signed and current
|
|
- admin/web ingress role is explicitly assigned to authorized node(s)
|
|
- TLS/certificate policy is valid
|
|
- node health and heartbeat are current
|
|
- required role coverage exists for the intended cluster operations
|
|
|
|
During cluster split/seed workflows, a temporary node may participate in more
|
|
than one cluster only through isolated memberships, identities, certificates,
|
|
tokens, storage namespaces, and policies. Removing that seed node from the new
|
|
cluster is safe only after the new cluster has its own healthy node set,
|
|
snapshots, role coverage, and admin ingress.
|
|
|
|
### Platform Admin Console MVP
|
|
|
|
The first admin panel should be for platform owner/admin only.
|
|
|
|
Required pages:
|
|
|
|
- Overview
|
|
- Clusters
|
|
- Nodes
|
|
- Join Requests
|
|
- Node Detail
|
|
- Role Assignments
|
|
- Service Workloads
|
|
- Health and Alerts
|
|
- Audit
|
|
- Trust and Certificates
|
|
|
|
Overview should show:
|
|
|
|
- cluster count
|
|
- node count
|
|
- nodes by health
|
|
- pending join requests
|
|
- role coverage gaps
|
|
- service workload health
|
|
- risky states
|
|
|
|
Cluster page should show:
|
|
|
|
- cluster status
|
|
- node membership
|
|
- enabled services
|
|
- health state
|
|
- partition warnings
|
|
- role coverage
|
|
|
|
Node page should show:
|
|
|
|
- identity
|
|
- membership
|
|
- health
|
|
- capabilities
|
|
- assigned roles
|
|
- running services
|
|
- last heartbeat
|
|
- version
|
|
- update state
|
|
- audit timeline
|
|
|
|
Join Requests page should allow:
|
|
|
|
- review node facts
|
|
- compare requested capabilities
|
|
- approve
|
|
- reject
|
|
- revoke token
|
|
|
|
Role Assignments page should allow:
|
|
|
|
- assign role to node
|
|
- restrict role to organization if applicable
|
|
- disable role
|
|
- audit role changes
|
|
|
|
### Multi-Cluster Admin
|
|
|
|
Multi-cluster administration should be platform-owner only.
|
|
|
|
It should support:
|
|
|
|
- list clusters
|
|
- compare cluster health
|
|
- inspect node distribution
|
|
- inspect service placement
|
|
- inspect cluster partitions
|
|
- inspect capacity and load
|
|
- detect missing role coverage
|
|
- manage cluster-scoped trust
|
|
|
|
Important:
|
|
No organization should see the full multi-cluster topology.
|
|
|
|
Multi-cluster trust boundaries:
|
|
|
|
- platform may operate multiple clusters
|
|
- clusters do not automatically trust each other
|
|
- clusters do not form one shared mesh by default
|
|
- cross-cluster routing requires explicit trust and policy
|
|
- a node may participate in multiple clusters only through isolated memberships
|
|
- cluster-scoped identities, certificates, tokens, storage namespaces, and policies are required
|
|
- platform owner may aggregate visibility across clusters through audited control-plane views
|
|
- organization admins see only authorized clusters, resources, services, and safe status
|
|
|
|
### Platform Admin Access Security
|
|
|
|
Platform owner/admin access must be risk-based.
|
|
|
|
Trusted or accredited devices may receive lower-friction access, but unknown
|
|
devices require stronger checks.
|
|
|
|
Required controls:
|
|
|
|
- MFA / 2FA policy
|
|
- device trust state
|
|
- session risk controls
|
|
- step-up authentication for high-risk actions
|
|
- audit for all high-risk actions
|
|
|
|
High-risk actions:
|
|
|
|
- cluster trust changes
|
|
- node approval
|
|
- role assignment
|
|
- partition promotion
|
|
- cross-cluster trust
|
|
- secrets access
|
|
- update policy changes
|
|
|
|
### Future Organization Admin Panel
|
|
|
|
Organization admin UI should come later.
|
|
|
|
It should show only safe tenant-scoped objects:
|
|
|
|
- organization resources
|
|
- organization users
|
|
- organization policies
|
|
- active sessions
|
|
- allowed entry points
|
|
- allowed egress/service endpoints
|
|
- safe VPN/connector status
|
|
- audit events for that organization
|
|
|
|
It must not show:
|
|
|
|
- full mesh topology
|
|
- other organizations
|
|
- internal core routing state
|
|
- platform-level node trust material
|
|
- unrelated cluster internals
|
|
|
|
## Service Visibility Rules
|
|
|
|
Organizations see only what policy allows:
|
|
|
|
- resources
|
|
- enabled services
|
|
- allowed ingress endpoints
|
|
- allowed egress/service endpoints
|
|
- safe status
|
|
|
|
Organizations do not see:
|
|
|
|
- intermediate core mesh nodes
|
|
- platform-owned private topology
|
|
- other tenants
|
|
- cluster-wide trust configuration
|
|
|
|
Different services may expose different ingress and egress points.
|
|
|
|
## Partition and Split-Brain Behavior
|
|
|
|
Disconnected minority segments must not:
|
|
|
|
- approve join requests
|
|
- issue node certificates
|
|
- assign roles
|
|
- change cluster policy
|
|
- rotate trust roots
|
|
|
|
Disconnected segments may continue already-running services only when safe and policy allows it.
|
|
|
|
Only platform owner authority should promote, merge, split, or recover partitions.
|
|
|
|
This document does not define final consensus mechanics. It only establishes that cluster authority and partition behavior must be modeled before production mesh changes.
|
|
|
|
## Implementation Stages
|
|
|
|
### Stage C1: Backend Cluster and Node Model Foundation
|
|
|
|
Status: implemented and verified. Report: `artifacts/c1-cluster-node-foundation-report.md`.
|
|
|
|
Goal:
|
|
Create the explicit backend data model and service boundaries for clusters, join requests, role assignments, node identity, heartbeats, and audit.
|
|
|
|
Allowed:
|
|
|
|
- migrations
|
|
- repositories
|
|
- service interfaces
|
|
- platform-admin-only API skeletons
|
|
- tests
|
|
- docs
|
|
|
|
Not allowed:
|
|
|
|
- mesh runtime
|
|
- VPN runtime
|
|
- node-agent runtime rewrite
|
|
- admin UI implementation
|
|
- RDP behavior changes
|
|
|
|
### Stage C2: Node Enrollment API
|
|
|
|
Status: implemented and verified. Report: `artifacts/c2-node-enrollment-hardening-report.md`.
|
|
|
|
Goal:
|
|
Implement production enrollment semantics.
|
|
|
|
Includes:
|
|
|
|
- join token creation
|
|
- join request creation
|
|
- approval/rejection
|
|
- certificate metadata boundary
|
|
- node activation/revocation
|
|
- audit
|
|
|
|
### Stage C3: Native Node-Agent MVP
|
|
|
|
Status: implemented and verified. Report: `artifacts/c3-rap-node-agent-mvp-report.md`.
|
|
|
|
Goal:
|
|
Create native `rap-node-agent` MVP.
|
|
|
|
Includes:
|
|
|
|
- enroll
|
|
- heartbeat
|
|
- capability report
|
|
- desired role polling
|
|
- service status report
|
|
|
|
No mesh packet routing yet.
|
|
|
|
### Stage C4: Platform Admin Console MVP
|
|
|
|
Status: implemented and build-verified. Report: `artifacts/c4-platform-admin-console-report.md`.
|
|
|
|
Goal:
|
|
Admin UI for platform operators.
|
|
|
|
Includes:
|
|
|
|
- clusters
|
|
- nodes
|
|
- join requests
|
|
- roles
|
|
- service health
|
|
- audit
|
|
|
|
### Stage C5: Service Workload Supervision Contract
|
|
|
|
Status: implemented and verified. Report: `artifacts/c5-service-workload-supervision-contract-report.md`.
|
|
|
|
Goal:
|
|
Node-agent can start, stop, and monitor service workloads based on role assignment.
|
|
|
|
C19A adds the first bounded live service-supervision runtime proof on top of
|
|
that contract: node-agent can read node-scoped desired workloads without an
|
|
operator actor id, report built-in `core-mesh` and `fabric-listener` as running,
|
|
report native built-in `synthetic.echo` as running, and keep unsupported
|
|
production workloads degraded instead of pretending that their adapters exist.
|
|
The live smoke is `scripts/fabric/c19a-service-workload-supervision-smoke.ps1`.
|
|
|
|
C19B adds the Remote Workspace/RDP adapter-contract bridge without enabling RDP
|
|
payload traffic. A native `rdp-worker` desired workload with
|
|
`adapter_contract_probe=true` reports the remote-workspace channel map,
|
|
requires Fabric Service Channel, and marks backend relay as not steady-state.
|
|
The live smoke is
|
|
`scripts/fabric/c19b-remote-workspace-adapter-contract-smoke.ps1`.
|
|
|
|
C19C wires Remote Workspace into service-channel lease issuance without
|
|
starting RDP traffic: route intents now accept `remote_workspace`, the lease
|
|
entry descriptor uses remote-workspace stream paths and frame-batch media type
|
|
instead of VPN packet paths, and the signed data-plane contract is present in
|
|
lease, authority payload, introspection, and lease maintenance. The live smoke
|
|
is `scripts/fabric/c19c-remote-workspace-service-channel-lease-smoke.ps1`.
|
|
|
|
C19D adds the Remote Workspace entry-node ingress skeleton. The node-agent
|
|
accepts a signed/introspected `remote_workspace` service-channel lease on
|
|
`remote-workspaces/{resource_id}/streams/{channel_class}`, validates service
|
|
class, channel class, selected entry node, and data-plane flow isolation, and
|
|
reports access telemetry. It intentionally returns a probe contract with
|
|
`payload_flow=validated_only` for empty control probes; non-empty RDP payloads are
|
|
rejected with `probe_only required`. This stage proves
|
|
the Fabric ingress contract without forwarding desktop frames yet. The live
|
|
smoke is `scripts/fabric/c19d-remote-workspace-entry-ingress-smoke.ps1`.
|
|
|
|
C19E adds the first Remote Workspace frame-batch contract probe across the
|
|
adapter/entry boundary. The `rdp-worker` adapter probe reports
|
|
`rap.remote_workspace_frame_batch.v1`; entry-node accepts only
|
|
`probe_only=true` frame batches, validates logical adapter channels and
|
|
directions, and returns `payload_flow=validated_probe_only`. Real desktop frame
|
|
delivery remains intentionally disabled until the service adapter runtime stage.
|
|
The live smoke is
|
|
`scripts/fabric/c19e-remote-workspace-frame-batch-contract-smoke.ps1`.
|
|
|
|
C19F adds the first local adapter-sink proof for that frame-batch contract.
|
|
Node-agent now keeps an in-memory `node_agent_rdp_worker_contract_probe` sink
|
|
for Remote Workspace frame probes and preserves it across mesh config refresh.
|
|
Entry-node delivers validated `probe_only=true` frame batches to that sink and
|
|
returns a `rap.remote_workspace_frame_batch_delivery.v1` receipt with
|
|
`payload_flow=delivered_probe_only`. This still does not enable production RDP
|
|
frame forwarding. The live smoke is
|
|
`scripts/fabric/c19f-remote-workspace-adapter-sink-smoke.ps1`.
|
|
|
|
C19G exposes the adapter-sink delivery proof through existing node-agent
|
|
visibility channels. The `rdp-worker` workload status payload now includes
|
|
`remote_workspace_adapter_sink`, and node telemetry includes
|
|
`remote_workspace_adapter_sink_report`, both carrying delivery count, latest
|
|
delivery sequence, channel class, frame count, and the probe-only/no-payload
|
|
boundary. The live smoke is
|
|
`scripts/fabric/c19g-remote-workspace-adapter-sink-telemetry-smoke.ps1`.
|
|
|
|
C19H locks down the Remote Workspace frame-batch guardrails before real adapter
|
|
runtime work begins. Unit and live smoke coverage now proves that entry-node
|
|
rejects `probe_only=false`, unknown logical channels, invalid channel
|
|
directions, service-class mismatch, channel-class mismatch, and unsupported
|
|
payload encoding, and that rejected batches do not produce adapter delivery.
|
|
The live smoke is
|
|
`scripts/fabric/c19h-remote-workspace-frame-guardrails-smoke.ps1`.
|
|
|
|
C19I adds the first bounded adapter handoff queue/ack proof for the same
|
|
probe-only path. The local `node_agent_rdp_worker_contract_probe` sink reports
|
|
queue capacity/depth plus accepted, dropped, and acked frame counts: with
|
|
capacity `8`, droppable display overflow accepts/acks `8` frames and drops `3`,
|
|
while reliable input overflow is rejected with backpressure and no delivery
|
|
receipt. The boundary still carries `payload_traffic=none`; this is queue
|
|
semantics for the future adapter runtime, not real RDP payload forwarding. The
|
|
live smoke is
|
|
`scripts/fabric/c19i-remote-workspace-adapter-queue-smoke.ps1`.
|
|
|
|
C19J makes those queue/backpressure signals operationally visible. The
|
|
`remote_workspace_adapter_sink` workload status payload and
|
|
`remote_workspace_adapter_sink_report` telemetry now include current queue
|
|
capacity/depth, cumulative accepted/dropped/acked frame counters,
|
|
`backpressure_count`, and the latest rejected batch metadata/reason. The live
|
|
smoke first produces the C19I droppable overflow plus reliable backpressure,
|
|
then waits until both workload status and telemetry show the delivery, dropped
|
|
total, and backpressure increment. The live smoke is
|
|
`scripts/fabric/c19j-remote-workspace-adapter-queue-telemetry-smoke.ps1`.
|
|
|
|
C19K introduces the probe-only adapter session boundary. Entry-node derives a
|
|
stable `adapter_session_id` from the service-channel lease/resource/route
|
|
context and passes it to the local `rdp-worker` adapter probe sink. Delivery
|
|
receipts, workload status, and telemetry now include `adapter_session_id`,
|
|
`adapter_runtime_id=node_agent_rdp_worker_contract_probe`, and
|
|
`session_state=probe_bound`, and rejected/backpressured batches retain the same
|
|
session identity. This is still not real RDP payload forwarding; it binds the
|
|
queue/ack/backpressure model to the future per-session adapter runtime. The
|
|
live smoke is
|
|
`scripts/fabric/c19k-remote-workspace-adapter-session-boundary-smoke.ps1`.
|
|
|
|
C19L adds the first lifecycle model to that probe-only adapter session. The
|
|
node-agent sink now tracks active sessions in memory with created/bound totals,
|
|
last activity timestamps, per-session delivery/backpressure/frame counters,
|
|
`current_session_lifecycle_state`, and idle expiry counters. A successful
|
|
droppable overflow binds the session as `probe_bound`; a reliable overflow keeps
|
|
the same `adapter_session_id` and moves the lifecycle state to `backpressure`
|
|
for diagnosis. Receipts expose session created/bound/last-activity timestamps
|
|
and per-session counters while `payload_traffic=none` remains enforced. The
|
|
live smoke is
|
|
`scripts/fabric/c19l-remote-workspace-adapter-session-lifecycle-smoke.ps1`.
|
|
|
|
C19M adds explicit probe-only adapter-session control. Node-agent exposes
|
|
`POST /mesh/v1/remote-workspace/adapter-sessions/{adapter_session_id}/control`
|
|
with `close`, `expire`, and `reset` actions, returning
|
|
`rap.remote_workspace_adapter_session_control.v1`. Workload status and telemetry
|
|
now include `session_control_total`, `session_closed_total`,
|
|
`session_reset_total`, and the latest control action/session/state, so sessions
|
|
can be ended deliberately instead of only by idle TTL. The live smoke creates a
|
|
Remote Workspace adapter session, closes it through the mesh control endpoint,
|
|
and waits until workload status and telemetry expose the close. The live smoke
|
|
is
|
|
`scripts/fabric/c19m-remote-workspace-adapter-session-control-smoke.ps1`.
|
|
|
|
C19N locks down the adapter-session control guardrails. Control requests now
|
|
reject unsupported actions, invalid `adapter_session_id` values, malformed JSON,
|
|
unknown active/terminal sessions, and overlong reasons without creating hidden
|
|
session state. Repeating `close` against an already closed terminal session is
|
|
idempotent: it reports `previous_state=closed` and does not increment
|
|
`session_closed_total` again, while still counting the control observation. The
|
|
live smoke verifies the negative cases plus first/repeated close visibility in
|
|
workload status and telemetry. The live smoke is
|
|
`scripts/fabric/c19n-remote-workspace-adapter-session-control-guardrails-smoke.ps1`.
|
|
|
|
C19O adds an immediate read-only adapter-session snapshot endpoint:
|
|
`GET /mesh/v1/remote-workspace/adapter-sessions?include_terminal=true&limit=N`.
|
|
It returns `rap.remote_workspace_adapter_session_snapshot.v1` with active
|
|
sessions, terminal sessions when requested, per-session lifecycle state,
|
|
activity/backpressure timestamps, frame counters, and runtime identity. This
|
|
lets operators inspect adapter-session state directly from node-agent without
|
|
waiting for heartbeat, workload status, or telemetry propagation. The live smoke
|
|
checks active-session visibility, close transition into terminal snapshot, and
|
|
invalid snapshot limit rejection. The live smoke is
|
|
`scripts/fabric/c19o-remote-workspace-adapter-session-snapshot-smoke.ps1`.
|
|
|
|
C19P adds the first adapter-runtime handoff mailbox contract. Each active
|
|
probe-only adapter session now owns a bounded in-memory mailbox that receives
|
|
`frame_batch_probe_delivered` and `backpressure` events with frame counts,
|
|
channel/resource/route context, and sequence numbers. Node-agent exposes
|
|
`GET /mesh/v1/remote-workspace/adapter-sessions/{adapter_session_id}/mailbox`
|
|
with optional `drain=true`, and session snapshots/workload reports expose
|
|
mailbox depth/enqueued/drained/dropped counters. This is the handoff surface a
|
|
real `rdp-worker` runtime can consume next; payload forwarding is still disabled.
|
|
The live smoke verifies read, drain, post-drain empty state, and snapshot
|
|
counters. The live smoke is
|
|
`scripts/fabric/c19p-remote-workspace-adapter-runtime-mailbox-smoke.ps1`.
|
|
|
|
C19Q hardens the mailbox handoff. Invalid IDs, unknown sessions, and invalid
|
|
limits are rejected before state mutation, and bounded `drain=true&limit=N`
|
|
reads remove only the returned event slice while preserving remaining depth for
|
|
the next poll. The bounded mailbox drops oldest events once capacity is reached,
|
|
and a closed adapter session no longer exposes an active runtime mailbox. The
|
|
live smoke verifies negative cases, drop-oldest pressure, partial drain, and
|
|
closed-session rejection. The live smoke is
|
|
`scripts/fabric/c19q-remote-workspace-adapter-mailbox-guardrails-smoke.ps1`.
|
|
|
|
C19R adds bounded long-poll ergonomics to the same node-local mailbox endpoint.
|
|
`wait_ms` lets an adapter runtime wait briefly for the next event without hot
|
|
polling, and responses make empty/timeout state explicit with `empty`,
|
|
`waited`, `wait_timeout`, and `wait_ms`. The live smoke proves empty timeout and
|
|
wake-on-delayed-event behavior while keeping the path probe-only. The live smoke
|
|
is `scripts/fabric/c19r-remote-workspace-mailbox-long-poll-smoke.ps1`.
|
|
|
|
C19S makes mailbox consumer behavior visible in diagnostics. Workload status and
|
|
node telemetry now expose `mailbox_read_total`, `mailbox_wait_total`,
|
|
`mailbox_wait_timeout_total`, `mailbox_empty_read_total`, and last mailbox read
|
|
metadata; active session snapshots carry the same per-session counters while a
|
|
session remains active. The live smoke proves C19R traffic is reflected in both
|
|
workload status and telemetry. The live smoke is
|
|
`scripts/fabric/c19s-remote-workspace-mailbox-telemetry-smoke.ps1`.
|
|
|
|
C19T adds the node-local consumer cursor contract for that mailbox. Consumers
|
|
can pass `consumer_id` plus optional `ack_sequence` to receive explicit
|
|
checkpoint, ack, lag, read, and ack counters without draining mailbox state.
|
|
The probe sink stores bounded per-session consumer state and reports aggregate
|
|
and current-session consumer telemetry through workload status and heartbeat
|
|
telemetry. The live smoke is
|
|
`scripts/fabric/c19t-remote-workspace-mailbox-consumer-checkpoint-smoke.ps1`.
|
|
|
|
C19U adds lifecycle visibility and reset guardrails to the same cursor state.
|
|
Mailbox consumers can pass `reset_consumer=true` with a valid `consumer_id` to
|
|
clear their checkpoint/ack state before the current read is recorded. Mailbox
|
|
responses now expose consumer count/capacity, created/reset/evicted flags, and
|
|
consumer timestamps, while diagnostics add reset and eviction counters. The
|
|
live smoke is
|
|
`scripts/fabric/c19u-remote-workspace-mailbox-consumer-lifecycle-smoke.ps1`.
|
|
|
|
C19V adds read-only inspection for active mailbox consumer cursors. The
|
|
node-local
|
|
`GET /mesh/v1/remote-workspace/adapter-sessions/{adapter_session_id}/mailbox/consumers`
|
|
endpoint returns bounded cursor snapshots with consumer ids, checkpoint and ack
|
|
sequences, lag, totals, and timestamps. It is verified as read-only: inspection
|
|
does not increment mailbox reads, ack totals, reset counters, or drain mailbox
|
|
events. The live smoke is
|
|
`scripts/fabric/c19v-remote-workspace-mailbox-consumer-snapshot-smoke.ps1`.
|
|
|
|
C19W adds cursor-aware resume reads to the mailbox endpoint. Consumers can pass
|
|
`after_sequence` to receive only mailbox events newer than their checkpoint;
|
|
responses include `skipped_count` and `returned_count`, and long-poll waits for
|
|
newer-than-checkpoint events. The endpoint rejects `after_sequence` with
|
|
`drain=true`, preserving the non-destructive resume contract. The live smoke is
|
|
`scripts/fabric/c19w-remote-workspace-mailbox-after-sequence-smoke.ps1`.
|
|
|
|
C19X adds consumer-aware resume convenience. Mailbox reads with `consumer_id`
|
|
can pass `resume_from=ack` or `resume_from=checkpoint`; the node-agent resolves
|
|
the stored cursor to `after_sequence` before reading and returns
|
|
`resume_from`/`resume_sequence` in the response. The guardrails reject mixing
|
|
resume with manual `after_sequence`, drain, reset, missing consumers, or invalid
|
|
cursor names. The live smoke is
|
|
`scripts/fabric/c19x-remote-workspace-mailbox-consumer-resume-smoke.ps1`.
|
|
|
|
C19Y adds resume telemetry to workload status and heartbeat reports. Operators
|
|
can now see resume read totals, after-sequence read totals, returned/skipped
|
|
totals, and the last resume cursor, sequence, consumer, returned count, and
|
|
skipped count. Session snapshots also expose per-session resume counters. The
|
|
live smoke is
|
|
`scripts/fabric/c19y-remote-workspace-mailbox-resume-telemetry-smoke.ps1`.
|
|
|
|
C19Z adds adapter-runtime readiness diagnostics. Sink reports now include
|
|
`adapter_runtime_readiness`, a compact probe-only object with ready status,
|
|
diagnostic state, session lifecycle, mailbox depth, consumer cursor, resume
|
|
cursor, lag, and returned/skipped counts. The live smoke is
|
|
`scripts/fabric/c19z-remote-workspace-adapter-readiness-smoke.ps1`.
|
|
|
|
C19Z1 adds read-only handoff preflight for mailbox consumers. The endpoint
|
|
`/mailbox/preflight` accepts `consumer_id` and `resume_from=ack|checkpoint`,
|
|
then reports the expected next event window without mailbox reads, drains, acks,
|
|
or consumer cursor mutation. The live smoke is
|
|
`scripts/fabric/c19z1-remote-workspace-mailbox-preflight-smoke.ps1`.
|
|
|
|
C19Z2 adds telemetry for mailbox preflight checks. Workload status and heartbeat
|
|
reports now expose preflight totals, ack/checkpoint split counters, and the last
|
|
preflight cursor/window fields so diagnostics can distinguish handoff checks
|
|
from mailbox reads. The live smoke is
|
|
`scripts/fabric/c19z2-remote-workspace-mailbox-preflight-telemetry-smoke.ps1`.
|
|
|
|
C19Z3 adds stale-cursor diagnostics to mailbox preflight. If a consumer cursor
|
|
falls behind retained mailbox events after bounded-mailbox drops, preflight
|
|
reports retained sequence bounds, `stale_cursor`, `diagnostic_state`, and
|
|
`missing_dropped_count`; the latest stale state is also visible in telemetry and
|
|
readiness diagnostics. The live smoke is
|
|
`scripts/fabric/c19z3-remote-workspace-mailbox-stale-preflight-smoke.ps1`.
|
|
|
|
C19Z4 adds action hints to mailbox preflight diagnostics. Stale cursor gaps now
|
|
return `recommended_action=reset_consumer_and_resync` plus hints to reset the
|
|
consumer cursor, request full adapter resync, and resume from checkpoint after
|
|
resync. The live smoke is
|
|
`scripts/fabric/c19z4-remote-workspace-mailbox-preflight-action-hints-smoke.ps1`.
|
|
|
|
C19Z5 adds provenance for the selected preflight action. Responses, telemetry,
|
|
and readiness diagnostics include `action_reason` and structured
|
|
`action_context` with cursor, retained sequence bounds, dropped/missing counts,
|
|
and expected window values. The live smoke is
|
|
`scripts/fabric/c19z5-remote-workspace-mailbox-preflight-provenance-smoke.ps1`.
|
|
|
|
C19Z6 adds the operator-facing summary for mailbox preflight. Responses,
|
|
telemetry, and readiness diagnostics include `operator_summary` plus compact
|
|
`operator_summary_fields` for the diagnostic state, selected action, reason,
|
|
cursor, retained bounds, and expected window counters. The live smoke is
|
|
`scripts/fabric/c19z6-remote-workspace-mailbox-preflight-summary-smoke.ps1`.
|
|
|
|
C19Z7 adds machine-sortable operator severity for mailbox preflight. Responses,
|
|
telemetry, readiness diagnostics, and summary fields expose `operator_status`
|
|
and `operator_severity`, classifying ready windows, caught-up cursors, and
|
|
stale cursor gaps without parsing summary text. The live smoke is
|
|
`scripts/fabric/c19z7-remote-workspace-mailbox-preflight-severity-smoke.ps1`.
|
|
|
|
C19Z8 adds the grouped readiness rollup for mailbox preflight. The readiness
|
|
diagnostic keeps the flat fields and adds `last_preflight` with observed time,
|
|
cursor, counts, diagnostic state, action hints/provenance, operator summary,
|
|
status, severity, and summary fields. The live smoke is
|
|
`scripts/fabric/c19z8-remote-workspace-mailbox-preflight-rollup-smoke.ps1`.
|
|
|
|
C19Z9 adds retained-window detail to that preflight rollup. The grouped
|
|
`last_preflight` readiness object includes first/last retained sequence and
|
|
mailbox dropped total so stale cursor explanations are visible without opening
|
|
the raw preflight response. The live smoke is
|
|
`scripts/fabric/c19z9-remote-workspace-mailbox-preflight-retained-window-smoke.ps1`.
|
|
|
|
C19Z10 adds a structured remediation checklist to that rollup. The grouped
|
|
`last_preflight.remediation_checklist` entries expose required/satisfied
|
|
operator steps derived from action hints, including cursor reset, full adapter
|
|
resync, and resume after resync for stale cursor gaps. The live smoke is
|
|
`scripts/fabric/c19z10-remote-workspace-mailbox-preflight-checklist-smoke.ps1`.
|
|
|
|
C19Z11 adds checklist status and counts to that rollup. The grouped
|
|
`last_preflight` readiness object exposes `remediation_checklist_status` and
|
|
total/required/satisfied/pending counts for admin UI summaries. The live smoke
|
|
is
|
|
`scripts/fabric/c19z11-remote-workspace-mailbox-preflight-checklist-status-smoke.ps1`.
|
|
|
|
C19Z12 adds session-level preflight operator status/severity counters.
|
|
Readiness exposes status and severity count maps, mirrored in `last_preflight`,
|
|
so repeated resync-required/warn preflights are visible without retaining a
|
|
history log. The live smoke is
|
|
`scripts/fabric/c19z12-remote-workspace-mailbox-preflight-status-counts-smoke.ps1`.
|
|
|
|
C19Z13 adds compact preflight attention status on top of those counters.
|
|
Readiness and `last_preflight` expose `preflight_attention_status` so admin UI
|
|
can sort clean, attention-needed, and repeated-resync sessions without
|
|
interpreting count maps. The live smoke is
|
|
`scripts/fabric/c19z13-remote-workspace-mailbox-preflight-attention-smoke.ps1`.
|
|
|
|
C19Z14 proves the repeated-resync attention branch. Unit and live smoke coverage
|
|
perform multiple stale preflights on one active adapter session and verify
|
|
`preflight_attention_status=repeated_resync_required` with repeated
|
|
resync-required/warn counters. The live smoke is
|
|
`scripts/fabric/c19z14-remote-workspace-mailbox-preflight-repeated-attention-smoke.ps1`.
|
|
|
|
C19Z15 adds the preflight attention reason. Readiness and `last_preflight`
|
|
expose `preflight_attention_reason` beside the attention status, explaining
|
|
clean, attention-needed, and repeated-resync states without UI-side counter
|
|
parsing. The live smoke is
|
|
`scripts/fabric/c19z15-remote-workspace-mailbox-preflight-attention-reason-smoke.ps1`.
|
|
|
|
C19Z16 completes focused proof coverage for those attention reasons. Unit tests
|
|
cover clean, single-resync, repeated-resync, and no-preflight mappings; live
|
|
smoke proves the single stale-preflight reason. The live smoke is
|
|
`scripts/fabric/c19z16-remote-workspace-mailbox-preflight-attention-reason-coverage-smoke.ps1`.
|
|
|
|
C19Z17 adds the preflight diagnostics contract marker. The readiness
|
|
`last_preflight` rollup includes `diagnostics_schema_version` and
|
|
`diagnostics_contract` entries for retained-window, remediation-checklist,
|
|
attention, and operator-count fields, allowing UI rendering to be gated safely.
|
|
The live smoke is
|
|
`scripts/fabric/c19z17-remote-workspace-mailbox-preflight-contract-smoke.ps1`.
|
|
|
|
C19Z18 adds boolean diagnostics feature flags to the same preflight rollup.
|
|
`last_preflight.diagnostics_features` now exposes retained-window,
|
|
remediation-checklist, attention, and operator-count support directly, so admin
|
|
UI and automation can gate each diagnostics group without scanning the contract
|
|
list. The live smoke is
|
|
`scripts/fabric/c19z18-remote-workspace-mailbox-preflight-feature-flags-smoke.ps1`.
|
|
|
|
C19Z19 proves compatibility between the two diagnostics contract forms. Unit
|
|
coverage and live smoke verify that workload and telemetry reports expose both
|
|
the string `diagnostics_contract` entries and matching boolean
|
|
`diagnostics_features` flags for every preflight diagnostics group. The live
|
|
smoke is
|
|
`scripts/fabric/c19z19-remote-workspace-mailbox-preflight-contract-compatibility-smoke.ps1`.
|
|
|
|
C19Z20 proves the no-preflight readiness shape. Before any mailbox preflight is
|
|
observed, active adapter sessions expose `preflight_attention_status=unknown`,
|
|
`preflight_attention_reason=no_preflight_observed`, zero session preflight
|
|
count, and no grouped `last_preflight` rollup. The live smoke is
|
|
`scripts/fabric/c19z20-remote-workspace-mailbox-preflight-absence-smoke.ps1`.
|
|
|
|
C19Z21 proves the no-active-session readiness shape. After closing the active
|
|
adapter session, readiness exposes idle/not-ready state, zero active sessions,
|
|
no active `adapter_session_id`, no grouped `last_preflight`, and terminal
|
|
`last_session_state=closed` from the terminal-session ledger. The live smoke is
|
|
`scripts/fabric/c19z21-remote-workspace-no-active-session-readiness-smoke.ps1`.
|
|
|
|
C19Z22 proves terminal-state readiness for `expire` and `reset` controls. The
|
|
same no-active-session readiness shape now reports
|
|
`last_session_state=expired` or `last_session_state=reset` from the
|
|
terminal-session ledger. The live smoke is
|
|
`scripts/fabric/c19z22-remote-workspace-terminal-state-readiness-smoke.ps1`.
|
|
|
|
C19Z23 adds grouped terminal-session summary metadata to no-active-session
|
|
readiness. `terminal_session_summary` carries adapter session id, terminal
|
|
state, reason, and control timestamp so admin UI can render the terminal cause
|
|
without stitching flat fields. The live smoke is
|
|
`scripts/fabric/c19z23-remote-workspace-terminal-session-summary-smoke.ps1`.
|
|
|
|
C19Z24 adds the terminal-session summary contract marker. The grouped summary
|
|
now carries schema version
|
|
`rap.remote_workspace_adapter_terminal_session_summary.v1` and a
|
|
summary-contract field list for explicit UI gating. The live smoke is
|
|
`scripts/fabric/c19z24-remote-workspace-terminal-summary-contract-smoke.ps1`.
|
|
|
|
C19Z25 adds boolean `summary_features` to the same grouped terminal-session
|
|
summary, covering adapter session id, state, reason, and control timestamp. The
|
|
live smoke is
|
|
`scripts/fabric/c19z25-remote-workspace-terminal-summary-features-smoke.ps1`.
|
|
|
|
C19Z26 proves compatibility between `summary_contract` and `summary_features`
|
|
for the grouped terminal-session summary in workload and telemetry reports. The
|
|
live smoke is
|
|
`scripts/fabric/c19z26-remote-workspace-terminal-summary-compatibility-smoke.ps1`.
|
|
|
|
C19Z27 proves the absence shape for terminal-session summary. Before any adapter
|
|
session or terminal history exists, readiness reports `waiting_for_session` and
|
|
does not include `terminal_session_summary`. The live smoke is
|
|
`scripts/fabric/c19z27-remote-workspace-terminal-summary-absence-smoke.ps1`.
|
|
|
|
Includes:
|
|
|
|
- container/native workload contract
|
|
- service instance status
|
|
- version reporting
|
|
- logs/health hooks
|
|
|
|
### Stage C6: Mesh Control-Plane Preparation
|
|
|
|
Status: implemented and verified. Report: `artifacts/c6-mesh-control-plane-preparation-report.md`.
|
|
|
|
Goal:
|
|
Prepare mesh routing state without carrying production traffic.
|
|
|
|
Includes:
|
|
|
|
- link inventory
|
|
- route intent
|
|
- node reachability
|
|
- QoS policy model
|
|
|
|
### Stage C7: Mesh MVP
|
|
|
|
Status: skeleton implemented and verified. Report: `artifacts/c7-mesh-mvp-skeleton-report.md`.
|
|
|
|
Goal:
|
|
First secure node-to-node overlay path.
|
|
|
|
Includes:
|
|
|
|
- mTLS node-to-node
|
|
- basic relay skeleton
|
|
- route health
|
|
- no VPN yet unless separately approved
|
|
|
|
### Stage C8: Multi-Cluster Hardening
|
|
|
|
Status: implemented and verified. Report: `artifacts/c8-multi-cluster-hardening-report.md`.
|
|
|
|
Goal:
|
|
Safe multi-cluster operations.
|
|
|
|
Includes:
|
|
|
|
- cluster authority state
|
|
- partition visibility
|
|
- cross-cluster admin views
|
|
- trust separation
|
|
|
|
### Stage C9: Organization Admin Console
|
|
|
|
Status: foundation implemented and verified. Report: `artifacts/c9-organization-admin-foundation-report.md`.
|
|
|
|
Goal:
|
|
Tenant-safe admin UI.
|
|
|
|
Includes:
|
|
|
|
- resources
|
|
- policies
|
|
- sessions
|
|
- allowed service endpoints
|
|
- organization audit
|
|
|
|
### Stage C10: Fabric Core Documentation and Config Distribution Design
|
|
|
|
Status: completed as documentation/planning stage. Report:
|
|
`artifacts/c10-fabric-core-config-distribution-design-report.md`.
|
|
|
|
Goal:
|
|
Consolidate Fabric Core terminology and prepare scoped cluster configuration
|
|
distribution design.
|
|
|
|
Scope:
|
|
|
|
- define signed scoped cluster snapshot model boundaries
|
|
- define node-local state boundaries
|
|
- define peer directory/cache boundaries
|
|
- define Fabric Storage / Config Storage role and non-goals
|
|
- define PostgreSQL source-of-truth versus distribution/cache boundaries
|
|
- define multi-cluster isolation boundaries
|
|
- define future implementation stages C11-C19
|
|
|
|
C10 prepares C11, C12, and C13. It did not implement mesh routing, VPN/IP
|
|
tunnel runtime, relay packet forwarding, RDP work, code, API changes, or
|
|
service workload execution.
|
|
|
|
### Stage C11: Signed Scoped Cluster Snapshot Model
|
|
|
|
Status: completed as documentation/planning stage. Report:
|
|
`artifacts/c11-signed-scoped-cluster-snapshot-model-report.md`.
|
|
|
|
Goal:
|
|
Define signed, versioned, scoped snapshots for node-local operation and
|
|
degraded-mode recovery.
|
|
|
|
### Stage C12: Node Local State Store
|
|
|
|
Status: completed as documentation/planning stage. Report:
|
|
`artifacts/c12-node-local-state-store-report.md`.
|
|
|
|
Goal:
|
|
Define bounded local state storage for identity, cluster membership, peer
|
|
cache, route cache, service assignment cache, health, partition/degraded state,
|
|
config versions, and pending update metadata.
|
|
|
|
### Stage C13: Config / Storage Service Foundation
|
|
|
|
Status: completed as documentation/planning stage. Report:
|
|
`artifacts/c13-fabric-storage-config-service-report.md`.
|
|
|
|
Goal:
|
|
Define Fabric Storage Service / Config Storage Service boundaries, replication
|
|
scope, failure-domain rules, and source-of-truth relationship to PostgreSQL.
|
|
|
|
### Stage C14: Peer Directory and Cache Model
|
|
|
|
Status: completed as documentation/planning stage. Report:
|
|
`artifacts/c14-peer-directory-cache-model-report.md`.
|
|
|
|
Goal:
|
|
Define peer directory shape, node-local peer cache, refresh cadence,
|
|
score-based peer selection inputs, and degraded recovery order.
|
|
|
|
### Stage C15: Fabric Routing Engine Skeleton
|
|
|
|
Status: completed as documentation/planning stage. Report:
|
|
`artifacts/c15-fabric-routing-engine-skeleton-report.md`.
|
|
|
|
Goal:
|
|
Introduce a safe compile/runtime boundary for future route selection without
|
|
carrying production mesh traffic.
|
|
|
|
### Stage C16: Secure Node-to-Node Channel Lifecycle
|
|
|
|
Status: completed as documentation/planning stage. Report:
|
|
`artifacts/c16-secure-node-to-node-channel-lifecycle-report.md`.
|
|
|
|
Goal:
|
|
Define authenticated node-to-node channel lifecycle, certificate validation,
|
|
health, reconnection, and channel authorization.
|
|
|
|
### Stage C17: Mesh Routing Runtime
|
|
|
|
Status: planning completed. Report:
|
|
`artifacts/c17-mesh-routing-runtime-implementation-plan-report.md`.
|
|
C17A synthetic runtime skeleton is implemented and test-proven. Report:
|
|
`artifacts/c17a-synthetic-mesh-runtime-skeleton-report.md`.
|
|
C17B route health and failover probe skeleton is implemented and test-proven.
|
|
Report: `artifacts/c17b-route-health-failover-probes-report.md`.
|
|
C17C relay semantic hardening skeleton is implemented and test-proven. Report:
|
|
`artifacts/c17c-relay-semantic-hardening-report.md`.
|
|
C17D non-production test-service path experiment is implemented and
|
|
test-proven. Report:
|
|
`artifacts/c17d-non-production-test-service-path-report.md`.
|
|
C17E live node-to-node synthetic HTTP transport skeleton is implemented and
|
|
smoke-proven. Report:
|
|
`artifacts/c17e-live-node-to-node-synthetic-transport-report.md`.
|
|
C17F scoped synthetic peer/route config loading and route-health reporting is
|
|
implemented and smoke-proven. Report:
|
|
`artifacts/c17f-scoped-synthetic-route-config-report.md`.
|
|
C17G Control Plane scoped synthetic config read boundary is implemented and
|
|
test-proven. Report:
|
|
`artifacts/c17g-control-plane-scoped-synthetic-config-report.md`.
|
|
|
|
Goal:
|
|
Define the mesh routing runtime implementation plan only after identity,
|
|
enrollment, scoped config, local state, storage, peer cache, routing skeleton,
|
|
and secure node-to-node channel lifecycle stages are accepted.
|
|
|
|
First implementation step was C17A: synthetic fabric messages only,
|
|
feature-flagged, test topology only, no RDP/VPN/service traffic, and no
|
|
topology exposure to organizations.
|
|
|
|
C17D proved a bounded `synthetic.echo` test-service path over direct,
|
|
single-relay, and forced fallback routes. It does not authorize production
|
|
service traffic.
|
|
|
|
C17E proved the same synthetic-only route model over real local HTTP node
|
|
endpoints using a disabled-by-default `rap-node-agent` endpoint and a
|
|
`mesh-live-smoke` harness. It still does not authorize production mesh traffic,
|
|
RDP/VPN traffic, service workload traffic, or organization-visible topology.
|
|
|
|
C17F moved the C17E synthetic route model from manual debug JSON toward scoped
|
|
configuration by adding a node-local scoped config file boundary and synthetic
|
|
route-health reporting to the Control Plane mesh link endpoint. It still does
|
|
not authorize production mesh traffic.
|
|
|
|
C17G added the Control Plane read endpoint for node-scoped synthetic mesh config
|
|
and node-agent consumption of that config. It still does not authorize
|
|
production mesh traffic.
|
|
|
|
### Stage C18: VPN / IP Tunnel Service
|
|
|
|
Status: completed as documentation/planning stage. Report:
|
|
`artifacts/c18-vpn-ip-tunnel-service-target-design-report.md`.
|
|
|
|
Goal:
|
|
Define VPN/IP tunnel service as a cluster-managed service above Fabric Core and
|
|
Fabric Routing, not as node-local ad hoc configuration.
|
|
|
|
Result:
|
|
|
|
- target design: `docs/architecture/VPN_IP_TUNNEL_SERVICE_TARGET.md`
|
|
- defines `vpn_connection` as logical desired state
|
|
- defines single-active lease/fencing model
|
|
- defines control-plane versus data-plane boundary
|
|
- defines routing policy, QoS, security, credential distribution, audit, and
|
|
failure behavior
|
|
- defines future C18A-C18I stages
|
|
|
|
C18 did not implement VPN/IP tunnel runtime, TUN/TAP devices, packet routing,
|
|
mesh production traffic, RDP work, API changes, migrations, or service workload
|
|
execution.
|
|
|
|
### Stage C18A: VPN / IP Tunnel Control-Plane Data Model
|
|
|
|
Status: implemented and backend-test-proven. Report:
|
|
`artifacts/c18a-vpn-control-plane-data-model-report.md`.
|
|
|
|
Goal:
|
|
Add the durable control-plane source-of-truth model for future VPN/IP tunnel
|
|
service desired state without implementing VPN runtime.
|
|
|
|
Completed:
|
|
|
|
- `vpn_connections`
|
|
- `vpn_connection_allowed_nodes`
|
|
- `vpn_connection_route_policies`
|
|
- `vpn_connection_leases`
|
|
- backend models, repository methods, service methods, and platform-admin API
|
|
skeleton
|
|
- single-active lease boundary with PostgreSQL unique active lease protection
|
|
- recovery-admin-only lease fencing boundary
|
|
- tests for authorization, scope, desired-state gating, duplicate active lease
|
|
rejection, route policy validation, and allowed-node normalization
|
|
|
|
C18A did not implement VPN/IP tunnel runtime, TUN/TAP devices, packet routing,
|
|
host route/firewall manipulation, node-agent execution, production mesh
|
|
traffic, RDP work, Windows client changes, or data-plane behavior changes.
|
|
|
|
### Stage C18B: VPN / IP Tunnel Lease and Fencing Hardening
|
|
|
|
Status: implemented and backend-test-proven. Report:
|
|
`artifacts/c18b-vpn-lease-fencing-hardening-report.md`.
|
|
|
|
Goal:
|
|
Harden the future VPN/IP tunnel single-active control-plane ownership model
|
|
before any node-agent executes VPN work.
|
|
|
|
Completed:
|
|
|
|
- owner eligibility validation against active cluster membership
|
|
- owner validation against `vpn_connection` allowed-node policy
|
|
- owner validation against active `vpn-exit` or `vpn-connector` role assignment
|
|
- same-owner acquire idempotency
|
|
- different-owner active lease rejection
|
|
- idempotent release/fence behavior
|
|
- stale lease cleanup/reclamation service boundary
|
|
- audit events for lease renewal and stale lease expiration
|
|
- tests for wrong cluster, unauthorized owner, missing role, disabled
|
|
connection, expired lease renewal, duplicate active lease, and stale lease
|
|
audit behavior
|
|
|
|
C18B did not implement VPN/IP tunnel runtime, TUN/TAP devices, host
|
|
route/firewall manipulation, node-agent VPN execution, production mesh traffic,
|
|
RDP work, Windows client changes, or data-plane behavior changes.
|
|
|
|
### Stage C18C: VPN / IP Tunnel Node-Agent Desired-State Consumption
|
|
|
|
Status: implemented and backend-test-proven. Report:
|
|
`artifacts/c18c-vpn-node-agent-desired-state-report.md`.
|
|
|
|
Goal:
|
|
Expose scoped VPN/IP tunnel desired assignments to eligible node-agents and
|
|
allow node-agents to report observed assignment status without executing VPN
|
|
runtime.
|
|
|
|
Completed:
|
|
|
|
- node-agent desired-state API for scoped VPN assignments
|
|
- node-agent assignment status report API
|
|
- explicit observed status values: `not_started`, `assigned`,
|
|
`lease_required`, `blocked`, `unknown`
|
|
- PostgreSQL assignment status report and latest-status tables
|
|
- backend service/repository boundaries for scoped assignment visibility
|
|
- assignment visibility limited to eligible candidates or current active owner
|
|
- `credential_ref` withheld from node-agent assignment payloads; only
|
|
`has_credential_ref` is exposed for future resolver integration
|
|
- tests for unauthorized/invisible assignment rejection, allowed status values,
|
|
invalid status rejection, and non-platform-admin node-agent read path
|
|
|
|
C18C did not implement VPN/IP tunnel runtime, TUN/TAP devices, host
|
|
route/firewall/DNS/QoS manipulation, credential delivery, node-agent VPN
|
|
execution, production mesh traffic, RDP work, Windows client changes, or
|
|
data-plane behavior changes.
|
|
|
|
### Stage C19: Version Storage and Node Update / Rollback Foundation
|
|
|
|
Status: future stage. Architecture direction is documented in this file and
|
|
`docs/architecture/SECURE_ACCESS_FABRIC_TARGET.md`. No updater runtime is
|
|
implemented by this documentation.
|
|
|
|
Goal:
|
|
Define the durable release, artifact, node-agent update, health supervision,
|
|
rollback, and data-structure migration model before any production update
|
|
runtime is enabled.
|
|
|
|
Scope:
|
|
|
|
- define `version-storage` / `update-repository` as the signed release
|
|
artifact source for node-agent and service workloads
|
|
- define release channels: `stable`, `current`, and `candidate`
|
|
- define OS / architecture artifact variants under a single signed release
|
|
manifest
|
|
- define last-known-good stable release tracking and rollback eligibility
|
|
- define update-cache mirroring of approved artifacts without becoming a
|
|
source of truth
|
|
- define node-agent download, verification, staging, restart, health-check,
|
|
promote, rollback, and failure-reporting states
|
|
- define migration bundle metadata for PostgreSQL schema, service-local data,
|
|
node-local state, config/cache shards, and protocol/config schema changes
|
|
- define rollout policy boundaries for canary, staged rollout, pause, resume,
|
|
rollback, blocklist, and emergency pinning
|
|
|
|
Non-goals:
|
|
|
|
- no production updater runtime
|
|
- no automatic schema migration execution by node-agent for PostgreSQL
|
|
- no unsigned or unapproved artifacts
|
|
- no arbitrary code download
|
|
- no mesh, VPN/IP tunnel, RDP, client, or data-plane behavior changes
|
|
|
|
Rules:
|
|
|
|
- PostgreSQL remains authoritative for rollout policy, approvals, desired
|
|
versions, channel assignment, migration approval, and audit.
|
|
- Version Storage stores immutable signed artifacts, release manifests,
|
|
compatibility metadata, hashes, signatures, provenance, and migration
|
|
bundles.
|
|
- Update-cache nodes only mirror approved artifacts and may be invalidated or
|
|
rebuilt from Version Storage.
|
|
- Node-agent executes only signed, approved, scoped work assigned by policy.
|
|
- Node-agent must stop, restart, rollback, or fence local workloads according
|
|
to health checks and lease/policy state.
|
|
- Node-agent self-update requires stricter safeguards than workload updates:
|
|
staged replacement, watchdog, crash-safe rollback, and preservation of
|
|
control-plane/update-source reachability unless an approved break-glass
|
|
procedure exists.
|
|
- PostgreSQL schema migrations are orchestrated by the Control Plane release
|
|
process, not independently invented by a node.
|
|
- Service-local and node-local migrations may be executed by node-agent only
|
|
when declared in a signed release manifest and scoped to that node/workload.
|
|
|
|
## Historical Stage C1 Prompt
|
|
|
|
This prompt is retained for provenance only. Stages C1-C9 are now listed above
|
|
as implemented/verified or build-verified according to their reports.
|
|
|
|
Proceed with Stage C1 only.
|
|
|
|
Goal:
|
|
Implement backend cluster and node model foundation for the Secure Access Fabric.
|
|
|
|
Strict rules:
|
|
|
|
- do NOT implement mesh runtime
|
|
- do NOT implement VPN runtime
|
|
- do NOT change RDP runtime behavior
|
|
- do NOT build admin UI yet
|
|
- do NOT rewrite existing node-agent runtime
|
|
- do NOT expose full topology to organizations
|
|
- keep PostgreSQL as source of truth
|
|
- keep Redis for live coordination only
|
|
|
|
Scope:
|
|
|
|
1. Add explicit cluster model.
|
|
2. Add cluster membership model.
|
|
3. Add node join token model with hashed tokens only.
|
|
4. Add node join request model.
|
|
5. Add node identity/certificate metadata model.
|
|
6. Add node role assignment model.
|
|
7. Add node heartbeat/latest health model.
|
|
8. Add cluster audit events.
|
|
9. Preserve existing node tables with safe migration/backfill into a default cluster.
|
|
10. Add repository interfaces and PostgreSQL implementations.
|
|
11. Add service/usecase boundaries for platform-admin cluster and node management.
|
|
12. Add platform-admin-only API skeletons for clusters, nodes, join requests, and role assignments.
|
|
13. Add tests for cluster scoping, role authorization, join token hashing, and organization topology isolation.
|
|
14. Update documentation.
|
|
|
|
Deliver:
|
|
|
|
- migrations
|
|
- backend models/repositories/services
|
|
- platform-admin API skeleton
|
|
- tests
|
|
- docs update
|
|
- verification report
|
|
|
|
Historical note:
|
|
C2 was not to start until C1 was accepted. C1-C9 are now recorded above with
|
|
their current accepted statuses.
|
|
|
|
## Result / Decision
|
|
|
|
This document defines Fabric Core as the foundation beneath service adapters
|
|
and before mesh runtime. It clarifies peer discovery, score-based routing,
|
|
scoped configuration distribution, node-local state, Fabric Storage/Config
|
|
Storage, multi-cluster trust boundaries, and risk-based platform admin access.
|
|
It also hardens the boundaries that Redis is live-only, Fabric routing must not
|
|
depend on live backend availability, and Fabric Storage/Config Storage is not a
|
|
general-purpose distributed database or second source of truth.
|
|
|
|
Stage C10 through Stage C18 planning are completed as documentation/planning.
|
|
C17A, C17B, C17C, C17D, C17E, C17F, and C17G are implemented and test-proven
|
|
with synthetic traffic only. C18 defines the VPN/IP tunnel service target model but does not
|
|
authorize VPN/IP tunnel runtime. C18A adds the VPN/IP tunnel control-plane
|
|
data model and platform-admin skeleton only. C18B hardens single-active
|
|
lease/fencing semantics. C18C adds node-agent desired-state/status reporting
|
|
for scoped VPN assignments only. C19 Remote Workspace adapter probe layers are
|
|
still node-local and probe-only; through C19Z30, fresh no-session runtime
|
|
readiness exposes a grouped `no_session_summary` contract plus
|
|
`summary_features`, with compatibility proof across workload and telemetry,
|
|
while terminal-history readiness exposes `terminal_session_summary` and omits
|
|
`no_session_summary`; summary exclusivity is proven across fresh, active, and
|
|
terminal readiness states, and a compact readiness state matrix artifact exists
|
|
for admin/runtime handoff. C19Z34 records the explicit probe-to-runtime gates
|
|
and confirms Remote Workspace still has no production payload traffic. C19Z35
|
|
adds the disabled-by-default real-adapter supervision status scaffold without
|
|
enabling real adapter execution. C19Z36 proves that scaffold's env/status/
|
|
guardrail compatibility. C19Z37 adds sanitized config projection for the future
|
|
real adapter while still refusing activation and payload traffic. C19Z38 proves
|
|
that projection for both default/empty and requested config shapes. C19Z39 adds
|
|
an explicit blocked activation decision contract with required/missing gates.
|
|
C19Z40 adds a compact handoff report proving scaffold/projection/decision
|
|
alignment for requested and default node config.
|
|
C19Z41 adds explicit feature flags for those real-adapter supervision fields.
|
|
C19Z42 folds those feature flags into the compact handoff report for
|
|
admin/runtime handoff.
|
|
C19Z43 proves contract-probe precedence when real-adapter supervision is also
|
|
requested in desired workload config.
|
|
C19Z44 proves the real-adapter-only desired workload path remains degraded and
|
|
blocked.
|
|
C19Z45 adds a compact desired-workload mode matrix for probe-only,
|
|
real-adapter-only, and combined requested modes.
|
|
C19Z46 adds compatibility proof for the mode matrix row contract.
|
|
C19Z47 adds a disabled process-supervisor preconditions contract for future
|
|
external RDP worker supervision.
|
|
C19Z48 proves that contract across requested/default config shapes.
|
|
C19Z49 folds process-supervisor preconditions into the compact handoff report.
|
|
C19Z50 folds process-supervisor preconditions into the desired-workload mode
|
|
matrix.
|
|
C19Z51 proves the mode matrix v2 row contract.
|
|
C19Z52 adds a disabled process-health-probe contract for future external RDP
|
|
worker supervision.
|
|
C19Z53 proves that process-health-probe contract across requested/default
|
|
status forms.
|
|
C19Z54 folds process-health-probe visibility into the compact real-adapter
|
|
handoff report.
|
|
C19Z55 folds process-health-probe visibility into the desired-workload mode
|
|
matrix.
|
|
C19Z56 proves the mode matrix v3 row contract.
|
|
C19Z57 adds a compact disabled real-adapter readiness/handoff checklist.
|
|
C19Z58 proves the readiness/handoff summary and checklist contract.
|
|
C19Z59 adds a disabled real-adapter operator action map.
|
|
C19Z60 proves the disabled real-adapter operator action map contract.
|
|
C19Z61 adds a compact disabled real-adapter admin handoff bundle.
|
|
C19Z62 proves the disabled real-adapter admin handoff bundle contract.
|
|
C19Z63 adds compact disabled real-adapter admin handoff digest rows.
|
|
C19Z64 proves the disabled real-adapter admin handoff digest row contract.
|
|
C19Z65 adds a disabled real-adapter admin handoff digest rollup.
|
|
C19Z66 proves the disabled real-adapter admin handoff digest rollup contract.
|
|
C19Z67 adds a disabled real-adapter admin handoff full-chain summary.
|
|
C19Z68 proves the disabled real-adapter admin handoff full-chain summary
|
|
contract.
|
|
C19Z69 adds a disabled real-adapter admin handoff release marker.
|
|
C19Z70 proves the disabled real-adapter admin handoff release marker contract.
|
|
C19Z71 adds a final contract-only package index for the disabled real-adapter
|
|
admin handoff chain.
|
|
C19Z72 proves the final package index contract for the disabled real-adapter
|
|
admin handoff chain.
|
|
C19Z73 adds a contract-only runtime gate phase boundary for the next disabled
|
|
real-adapter preflight phase.
|
|
C19Z74 proves the runtime gate phase boundary contract.
|
|
C19Z75 adds a disabled real-adapter runtime gate preflight checklist with all
|
|
items still blocking runtime.
|
|
C19Z76 proves the disabled real-adapter runtime gate preflight checklist
|
|
contract.
|
|
C19Z77 adds a disabled real-adapter runtime gate preflight status summary.
|
|
C19Z78 proves the disabled real-adapter runtime gate preflight status summary
|
|
contract.
|
|
C19Z79 adds disabled real-adapter runtime gate preflight action hints.
|
|
C19Z80 proves the disabled real-adapter runtime gate preflight action hints
|
|
contract.
|
|
C19Z81 adds a disabled real-adapter runtime gate preflight operator handoff
|
|
bundle.
|
|
C19Z82 proves the disabled real-adapter runtime gate preflight operator handoff
|
|
bundle contract.
|
|
C19Z83 adds a disabled real-adapter runtime gate preflight release marker.
|
|
C19Z84 proves the disabled real-adapter runtime gate preflight release marker
|
|
contract.
|
|
C19Z85 adds a disabled real-adapter runtime gate preflight package index.
|
|
C19Z86 proves the disabled real-adapter runtime gate preflight package index
|
|
contract.
|
|
C19Z87 adds a disabled real-adapter runtime gate preflight closeout summary.
|
|
C19Z88 proves the disabled real-adapter runtime gate preflight closeout summary
|
|
contract.
|
|
C19Z89 starts the explicit real-adapter runtime gate enablement phase with a
|
|
contract-only request that remains blocked pending validation.
|
|
C19Z90 proves the explicit real-adapter runtime gate enablement request
|
|
contract.
|
|
C19Z91 adds contract-only operator confirmation validation while keeping the
|
|
runtime gate blocked pending remaining validations.
|
|
C19Z92 proves the operator confirmation validation contract.
|
|
C19Z93 adds contract-only binary validation while keeping the runtime gate
|
|
blocked pending remaining validations.
|
|
C19Z94 proves the binary validation contract.
|
|
C19Z95 adds contract-only permission validation while keeping the runtime gate
|
|
blocked pending remaining validations.
|
|
C19Z96 proves the permission validation contract.
|
|
C19Z97 adds contract-only supervisor validation while keeping the runtime gate
|
|
blocked pending remaining validations.
|
|
C19Z98 proves the supervisor validation contract.
|
|
C19Z99 adds contract-only health probe validation while keeping the runtime gate
|
|
blocked pending payload gate validation.
|
|
C19Z100 proves the health probe validation contract.
|
|
C19Z101 adds contract-only payload gate validation with no remaining required
|
|
validations while keeping runtime not enabled.
|
|
C19Z102 proves the payload gate validation contract.
|
|
C19Z103 adds the runtime gate validation closeout while keeping explicit
|
|
operator enablement required.
|
|
C19Z104 proves the runtime gate validation closeout contract.
|
|
C19Z105 adds an operator enablement readiness package while keeping runtime
|
|
disabled by default.
|
|
C19Z106 proves the operator enablement readiness package contract.
|
|
C19Z107 adds an operator enablement readiness release marker while keeping
|
|
runtime disabled by default.
|
|
C19Z108 proves the operator enablement readiness release marker contract.
|
|
C19Z109 adds an operator enablement readiness package index while keeping
|
|
runtime disabled by default.
|
|
C19Z110 proves the operator enablement readiness package index contract.
|
|
C19Z111 adds an operator readiness closeout summary while keeping runtime
|
|
disabled by default.
|
|
C19Z112 proves the operator readiness closeout summary contract.
|
|
C19Z113 adds an operator review decision request while keeping runtime disabled
|
|
by default.
|
|
C19Z114 proves the operator review decision request contract.
|
|
C19Z115 adds an operator decision status summary while keeping runtime disabled
|
|
by default.
|
|
C19Z116 proves the operator decision status summary contract.
|
|
C19Z117 adds an operator approval/rejection outcome contract with the outcome
|
|
not approved and runtime disabled by default.
|
|
C19Z118 proves the operator approval/rejection outcome contract.
|
|
C19Z119 adds an operator outcome closeout/reopen boundary while keeping runtime
|
|
disabled by default.
|
|
C19Z120 proves the operator outcome closeout/reopen boundary contract.
|
|
C19Z121 adds a not-approved outcome release marker while keeping runtime
|
|
disabled by default.
|
|
C19Z122 proves the not-approved outcome release marker contract.
|
|
C19Z123 adds a not-approved outcome package index while keeping runtime disabled
|
|
by default.
|
|
C19Z124 proves the not-approved outcome package index contract.
|
|
C19Z125 adds a not-approved outcome closeout summary while keeping runtime
|
|
disabled by default.
|
|
C19Z126 proves the not-approved outcome closeout summary contract.
|
|
C19Z127 adds a final not-approved outcome release marker while keeping runtime
|
|
disabled by default.
|
|
C19Z128 proves the final not-approved outcome release marker contract.
|
|
C19Z129 adds a final not-approved outcome package index/archive marker while
|
|
keeping runtime disabled by default.
|
|
C19Z130 proves the final not-approved outcome package index/archive marker
|
|
contract.
|
|
C19Z131 adds a not-approved outcome archive closeout manifest while keeping
|
|
runtime disabled by default.
|
|
C19Z132 proves the not-approved outcome archive closeout manifest contract.
|
|
C19Z133 adds a stopped-branch sentinel for the not-approved outcome while
|
|
keeping runtime disabled by default.
|
|
C19Z134 proves the not-approved outcome stopped-branch sentinel contract.
|
|
C19Z135 adds a no-continuation guard for the stopped not-approved outcome while
|
|
keeping runtime disabled by default.
|
|
C19Z136 proves the not-approved outcome no-continuation guard contract.
|
|
C19Z137 adds continuation block enforcement for the stopped not-approved
|
|
outcome while keeping runtime disabled by default.
|
|
C19Z138 proves the not-approved outcome continuation block enforcement
|
|
contract.
|
|
C19Z139 adds a continuation block audit record for the stopped not-approved
|
|
outcome while keeping runtime disabled by default.
|
|
C19Z140 proves the not-approved outcome continuation block audit record
|
|
contract.
|
|
C19Z141 adds a continuation block audit rollup for the stopped not-approved
|
|
outcome while keeping runtime disabled by default.
|
|
C19Z142 proves the not-approved outcome continuation block audit rollup
|
|
contract.
|
|
C19Z143 adds an operator stop summary for the stopped not-approved outcome
|
|
while keeping runtime disabled by default.
|
|
C19Z144 proves the not-approved outcome operator stop summary contract.
|
|
C19Z145 adds an operator stop handoff for the stopped not-approved outcome
|
|
while keeping runtime disabled by default.
|
|
C19Z146 proves the not-approved outcome operator stop handoff contract.
|
|
C19Z147 adds an operator stop handoff digest for the stopped not-approved
|
|
outcome while keeping runtime disabled by default.
|
|
C19Z148 proves the not-approved outcome operator stop handoff digest contract.
|
|
C19Z149 adds an operator stop status snapshot for the stopped not-approved
|
|
outcome while keeping runtime disabled by default.
|
|
C19Z150 proves the not-approved outcome operator stop status snapshot contract.
|
|
C19Z151 adds an operator stop status snapshot index for the stopped
|
|
not-approved outcome while keeping runtime disabled by default.
|
|
C19Z152 proves the not-approved outcome operator stop status snapshot index
|
|
contract.
|
|
C19Z153 adds an operator stop status catalog for the stopped not-approved
|
|
outcome while keeping runtime disabled by default.
|
|
C19Z154 proves the not-approved outcome operator stop status catalog contract.
|
|
C19Z155 adds an operator stop status catalog release marker for the stopped
|
|
not-approved outcome while keeping runtime disabled by default.
|
|
C19Z156 proves the not-approved outcome operator stop status catalog release
|
|
marker contract.
|
|
C19Z157 adds an operator stop status catalog package index for the stopped
|
|
not-approved outcome while keeping runtime disabled by default.
|
|
C19Z158 proves the not-approved outcome operator stop status catalog package
|
|
index contract.
|
|
C19Z159 adds an operator stop status catalog closeout summary for the stopped
|
|
not-approved outcome while keeping runtime disabled by default.
|
|
C19Z160 proves the not-approved outcome operator stop status catalog closeout
|
|
summary contract.
|
|
C19Z161 adds an operator stop status final archive marker for the stopped
|
|
not-approved outcome while keeping runtime disabled by default.
|
|
C19Z162 proves the not-approved outcome operator stop status final archive
|
|
marker contract.
|
|
C19Z163 adds an operator stop status final archive manifest for the stopped
|
|
not-approved outcome while keeping runtime disabled by default.
|
|
C19Z164 proves the not-approved outcome operator stop status final archive
|
|
manifest contract.
|
|
C19Z165 adds a terminal-complete marker for the stopped not-approved outcome
|
|
factory while keeping runtime disabled by default.
|
|
C19Z166 proves the not-approved outcome factory terminal-complete contract.
|
|
C20Z1 opens a new explicit real-adapter enablement request while keeping
|
|
runtime disabled by default.
|
|
C20Z2 proves the new explicit real-adapter enablement request contract.
|
|
C20Z3 adds the operator validation intake for the new explicit request while
|
|
keeping runtime disabled by default.
|
|
C20Z4 completes the operator validation checklist contract while keeping
|
|
runtime disabled by default.
|
|
C20Z5 closes the operator validation chain contract while keeping runtime
|
|
disabled by default.
|
|
C20Z6 proves the C20 stage terminal-complete contract.
|
|
Version Storage/Update
|
|
Repository and node-agent update/rollback foundation are not implemented by
|
|
this document. No RDP, data-plane, VPN runtime, production relay, production
|
|
mesh service traffic, node-agent VPN execution, host networking, service
|
|
workload runtime, or production updater behavior is implied by this document.
|