rdp-proxy/docs/architecture/CLUSTER_NODE_ADMIN_FOUNDATION.md

Cluster, Node, Mesh, and Admin Foundation

Status: target foundation plan plus staged implementation tracker. This document defines the next platform-core direction and implementation order. It is not an instruction to rewrite the current RDP MVP.

The current RDP work is paused by product decision. The proven RDP access core remains valuable and must not be broken. The next platform step is to build the cluster, node enrollment, mesh preparation, and platform administration foundation that the Secure Access Fabric will depend on.

Related documents:

• docs/architecture/SECURE_ACCESS_FABRIC_TARGET.md
• docs/architecture/DATA_PLANE_V1.md
• docs/architecture/WEB_INGRESS_AND_ADMIN_UI_MODEL.md
• docs/codex/CURRENT_STATUS.md

Purpose

The platform is a Secure Access Fabric, not only an RDP proxy. RDP is the first managed service adapter, but the long-term platform must support additional adapters and network services such as VNC, SSH, VPN/IP tunnel, internal web apps, file services, video/audio, update cache, and relay/entry nodes.

The lower platform foundation is the RAP Fabric Core: a distributed runtime layer above the host OS and below service adapters. It is not a real operating system. It is implemented through native rap-node-agent, control-plane contracts, signed scoped cluster snapshots, node-local state, and service supervision boundaries.

Before implementing mesh traffic, VPN runtime, or organization administration UI, the platform needs a clear foundation for:

• clusters
• host-level node identity
• node enrollment
• Fabric Core terminology and contracts
• scoped configuration distribution
• node-local state
• role assignment
• service workload placement
• platform-owner administration
• future organization administration
• safe multi-cluster operations

Current Foundation Inventory

The backend already contains an early platform-core foundation:

• nodes
• node_capabilities
• node_services
• node_update_policies
• node_partition_states
• node_agent_update_runs
• node management routes under /nodes
• node-agent routes under /node-agents

This is useful, but it is not sufficient as the final node and cluster model. Current gaps:

• no explicit first-class clusters
• no explicit cluster membership model
• no join request approval workflow
• no short-lived join token model
• no node certificate lifecycle model
• no clear separation between technical capability and authorized role assignment
• no platform admin console model
• no multi-cluster administration model
• no organization-safe visibility model for entry/egress/service endpoints
• node-agent registration is too direct for production trust boundaries

Vocabulary

Platform: The whole Secure Access Fabric installation operated by the platform owner.

Cluster: A bounded control and trust domain containing nodes, policies, service placements, certificates, health state, and routing state. A platform may operate multiple clusters.

Organization: A tenant. Organizations own or are granted access to resources, policies, sessions, services, and selected entry/egress points. Organizations must not see full internal mesh topology.

Fabric Core: The lower OS-like distributed runtime layer of the Secure Access Fabric. It runs above the host OS and is owned by native rap-node-agent plus control-plane contracts. It owns node identity, enrollment, local state, capability reporting, role assignment consumption, signed scoped configuration snapshots, update trust, and service supervision boundaries.

Node: A host-level identity. A node is not a Docker container. A node is managed by native rap-node-agent.

Node Agent: Native host process responsible for node identity, enrollment, certificates, health, role execution, service workload supervision, updates, recovery, and reporting.

Service Workload: A workload executed on a node. It may be native or containerized. Examples: rdp-worker, vnc-worker, entry-node, relay-node, file-storage-cache.

Capability: What a node can technically do. Example: can_run_rdp_worker.

Role Assignment: What a node is authorized to do in a cluster and, where relevant, for an organization. Capabilities are not permissions.

Join Request: A request from a node-agent to join a cluster, usually created with a short-lived join token and approved by a platform admin.

Entry Node: Accepts client connections and maps them into platform logical channels.

Core Mesh Node: Maintains secure overlay routing and QoS. It should remain protocol-neutral.

Egress / Service Node: Hosts service adapters or network exits that connect to target resources.

Service Adapter: Protocol-specific edge function, for example RDP Adapter, VNC Adapter, SSH Adapter, HTTP/internal app adapter, or video/audio adapter.

Remote Server/Desktop Access: Product-level managed access service consumed by Access Client. RDP, VNC, and SSH are internal protocol adapters selected from the organization resource protocol; they are not separate organization-facing cluster services.

Egress Pool: Organization-visible logical exit such as "Office Moscow" or "Datacenter Kazan". It may be backed by multiple internal nodes. Organizations reference egress pools, not concrete nodes or mesh topology.

Fabric Routing Engine: Logical fabric-layer component responsible for peer selection, route discovery, route scoring, failover, shortcut decisions, route cache management, topology hiding, and channel-aware routing. It is not a runtime implementation in the current phase.

Fabric Storage Service / Config Storage Service: Future scoped storage/distribution service for signed cluster snapshots, peer directories, policy snapshots, update metadata, and node-scoped configuration. PostgreSQL remains authoritative.

Version Storage / Update Repository: Logical artifact repository for signed platform, node-agent, and service workload releases. It stores release manifests, OS/architecture artifacts, stable/current/candidate channel metadata, hashes, signatures, compatibility metadata, and migration bundles. PostgreSQL remains authoritative for rollout policy, approvals, and audit.

Golden Rules

1. Node identity belongs to the native rap-node-agent, not to a container.
2. Containers are packaging and isolation boundaries only.
3. Capabilities are not permissions.
4. Role assignment must be explicit per cluster and, when needed, per organization.
5. Platform admins operate cluster/node trust and placement.
6. Organization admins manage only organization-visible resources, policies, sessions, and allowed service endpoints.
7. Organizations must not see intermediate core mesh topology.
8. Mesh routing must not be implemented before enrollment, identity, and role assignment are trustworthy.
9. Disconnected minority partitions must not add nodes or change policy.
10. PostgreSQL remains the source of truth; Redis/live state must be reconstructable.
11. RDP MVP behavior must remain preserved while platform-core work progresses.
12. Fabric Core comes before mesh runtime.
13. Service adapters must not own mesh routing logic.
14. Nodes must not store or learn more cluster, organization, secret, or route data than their role requires.
15. Config/storage services distribute scoped snapshots and caches; they are not a second source of truth.
16. Redis must not store durable topology, durable configuration, node identity, policy, or organization data.
17. Fabric routing decisions must not depend on live backend availability.
18. Fabric Storage / Config Storage must not become a general-purpose distributed database.
19. Version Storage stores signed artifacts and release manifests; it does not decide rollout policy and must not serve unsigned or unapproved artifacts.
20. Node-agent is the local supervisor for health, restart, update, and rollback of node services, but Control Plane owns rollout policy and durable schema migration orchestration.

Existing Node Management Semantics

The Platform Owner Console must treat a node as a concrete host-level identity, not as an abstract request for any free capacity. A cluster membership always connects a specific node_id to a specific cluster_id.

The same physical node may participate in multiple clusters only through isolated cluster memberships, certificates, tokens, storage namespaces, and policies. Role assignments and desired service workloads are cluster-scoped. This means a node may run entry-node for one cluster, have no ingress role in another cluster, and still report the same underlying host capabilities.

Node capabilities are reported by rap-node-agent as the technical upper bound of what the host can do. Capabilities are not permissions. The Control Plane must expose to each cluster only the roles and service workloads that are explicitly assigned for that cluster and, where relevant, for a specific organization.

Current owner-console operations for existing nodes are:

• view active-cluster nodes
• view all physical nodes visible to the platform owner
• inspect cluster membership state per node
• organize active-cluster nodes into hierarchical node groups
• inspect heartbeat, telemetry, reported services, and desired services
• assign cluster-scoped roles
• enable or disable cluster-scoped desired service workloads
• manage node participation in product-level functions such as Remote Server/Desktop Access, without exposing protocol adapters as separate organization-facing services
• enable node/cluster/organization test telemetry flags
• disable a node membership in the active cluster
• revoke a node identity as a high-risk platform-owner action

Creating a brand-new node is not a direct database insert. The future production flow remains installation of native rap-node-agent on the host, enrollment with a short-lived join token, platform-owner approval, then explicit role and service assignment. This enrollment/create-node UX is intentionally separate from existing-node management.

Node groups are cluster-scoped inventory folders, not node identities and not permissions. A group may contain child groups, and a node membership may be assigned to one group in that cluster. The same physical node may be in a different group, or no group, in another cluster. Groups are intended for large-scale operator usability: data centers, sites, customers, regions, environments, ownership boundaries, or maintenance rings. Authorization still comes from cluster membership, role assignment, organization policy, and service desired state, not from the visual group tree.

Remote Server/Desktop Access is managed as one product service. A node may be allowed to run this access service, while the concrete protocol adapter is chosen from the organization resource (rdp, vnc, ssh). Cluster admins should not configure separate organization-facing RDP/VNC/SSH services; those adapters are internal runtime implementations.

Fabric Core Foundation

Layer order:

1. Host OS
2. RAP Fabric Core
3. Secure Fabric Network
4. Service Runtime / Service Adapters
5. Access Clients / Admin UI

Fabric Core responsibilities:

• node identity
• node enrollment
• local node state
• capability reporting
• role assignment consumption
• signed scoped cluster snapshots
• update trust
• service workload supervision boundary

Fabric Core does not implement service protocols. RDP, VNC, SSH, VPN, video, and file transfer are services above the fabric. They consume node identity, role assignment, local state, routing, and policy from the Fabric Core.

Fabric Core must be trustworthy before mesh runtime traffic exists. Mesh routing, VPN/IP tunnel runtime, relay packet routing, and service workload execution must not be implemented before node identity, enrollment, trust, role assignment, and scoped configuration distribution are solid.

Peer Discovery and Routing Foundation

Nodes must not maintain active connections to all other nodes.

Each node maintains:

• active peers
• warm candidate peers
• cold / bootstrap peers

Recommended defaults:

• normal node: 3-5 active peers
• relay / core node: 8-20 active peers
• thin / mobile node: 1-3 active peers

Peer selection must be score-based, not latency-only.

Score inputs:

• latency
• packet loss
• reliability
• region distance
• node load
• bandwidth availability
• role suitability
• policy constraints
• trust level
• recent failure history

Service adapters request a destination node, resource target, egress node, or egress pool. The Fabric Routing Engine chooses the path. RDP/VNC/SSH/VPN/video adapters must not implement topology discovery, multi-hop route selection, shortcut creation, or cross-cluster routing policy.

Fabric routing uses node-local state, signed scoped snapshots, peer cache, and route cache. It must respect policy, organization scope, cluster boundaries, and partition/authority state. Service adapters must not select routes, discover peers, manage mesh connections, implement failover, implement shortcut logic, implement partition recovery, or implement cross-cluster routing policy.

Node-Local State and Scoped Configuration

rap-node-agent local state should include:

• node identity
• cluster membership
• signed scoped cluster snapshot
• peer cache
• route cache
• service assignment cache
• local health state
• partition / degraded state
• last applied config version
• pending update metadata

A node must not store full cluster topology unless its role requires it.

Configuration distribution is need-to-know:

• core mesh nodes receive neighbor/peer data, route policy, QoS policy, and cluster version
• ingress nodes receive allowed client entry policies, token validation config, and route entry data
• egress/service nodes receive assigned service configs, needed resource refs, connector refs, and service policy
• storage services receive assigned shard/scope data and replication metadata

Secrets are delivered only through approved resolvers and only when required at runtime. RDP credentials, organization user lists, unrelated storage shards, and full topology must not be distributed to nodes that do not need them.

Fabric Storage / Config Storage Foundation

Fabric Storage Service / Config Storage Service is a future foundation service, not a replacement for PostgreSQL.

Purpose:

• store and replicate scoped cluster configuration
• distribute signed snapshots
• keep frequently used data near services
• support local high-speed reads
• preserve configuration availability when some nodes disappear

Rules:

• not every node stores all data
• replication factor is policy-driven
• critical data is replicated across failure domains
• hot data is placed near services that use it
• organization and cluster isolation are mandatory
• PostgreSQL remains authoritative
• storage/config services are distribution/cache layers only
• storage/config services must not accept direct node writes as authoritative state
• storage/config services must not expose arbitrary query capabilities
• storage/config services must not store full cluster or organization data on every node

Data classes:

• platform global data
• cluster state
• organization data
• service config data
• realtime / lease state
• audit / event data
• artifacts / update data

Version Storage and Node Update Foundation

The platform includes a logical Version Storage / Update Repository service.

It stores approved release artifacts and metadata for:

• native rap-node-agent
• service workloads such as rdp-worker, entry-node, relay-node, file-storage-cache, update-cache, future VNC/SSH/VPN services
• OS / architecture-specific builds
• migration bundles for data structure changes

Release channels:

• stable: known-good version used for rollback
• current: normal intended version for the cluster/ring
• candidate: version staged for test/canary rollout

Rules:

• PostgreSQL is authoritative for rollout policy, channel assignment, approvals, target nodes/rings, and audit.
• Version Storage stores immutable signed artifacts and manifests.
• update-cache mirrors approved artifacts close to nodes but does not approve or invent versions.
• Node-agent downloads from authorized update-cache or Version Storage, verifies signatures and hashes, stages updates, runs health checks, and reports state.
• Node-agent may restart workloads and roll back to last stable / last known good when health checks fail.
• Node-agent self-update requires stronger safeguards than normal service workloads: A/B slots or equivalent staged replacement, watchdog/recovery, and rollback that survives process crash.

Data structure migrations:

• code updates and data structure updates are separate concerns in one signed release manifest
• migration bundles must declare source version, target version, compatibility, preflight checks, backup/snapshot requirements, rollback behavior, and whether downgrade is possible
• Control Plane/PostgreSQL schema migrations are orchestrated by the Control Plane release process
• service-local, cache-local, and node-local state migrations may be executed by node-agent only when included in an approved signed manifest
• failed migrations must leave the node rolled back, degraded, or fenced, never silently half-updated

Suggested update states:

• idle
• checking
• downloading
• verifying
• staging
• migrating
• starting
• health_checking
• promoting
• running
• rollback_required
• rolling_back
• rolled_back
• failed
• fenced

Target Data Model

The following entities should become the explicit foundation. Names are target names and may be adjusted during implementation if the existing schema requires compatibility.

Cluster

clusters

Purpose: First-class cluster records.

Key fields:

• id
• name
• slug
• status
• region
• metadata
• created_at
• updated_at

Cluster Membership

cluster_memberships

Purpose: Links nodes to clusters with membership state.

Key fields:

• cluster_id
• node_id
• membership_status
• joined_at
• last_seen_at
• metadata

Node

nodes

Purpose: Host-level identity record.

Existing nodes should be preserved and migrated safely. A default cluster backfill is acceptable for existing rows.

Target additions or refinements:

• explicit cluster membership through cluster_memberships
• enrollment status separated from runtime health
• identity/certificate state
• ownership and organization association only where applicable

Node Identity

node_identities

Purpose: Stores node public identity material and trust state.

Key fields:

• node_id
• public_key
• certificate_serial
• certificate_not_before
• certificate_not_after
• identity_status
• rotated_at
• revoked_at

Private keys must never be stored in the control plane.

Node Join Token

node_join_tokens

Purpose: Short-lived token created by platform admin to allow enrollment into a cluster.

Key fields:

• id
• cluster_id
• token_hash
• scope
• expires_at
• max_uses
• used_count
• created_by_user_id
• revoked_at

Raw join tokens must not be stored.

Node Join Request

node_join_requests

Purpose: Approval workflow for node enrollment.

Key fields:

• id
• cluster_id
• node_name
• node_fingerprint
• public_key
• reported_capabilities
• reported_facts
• requested_roles
• status
• reviewed_by_user_id
• reviewed_at
• approved_node_id
• rejection_reason

Default behavior should be manual approval.

Node Capability

node_capabilities

Purpose: Technical capabilities reported by node-agent and verified by platform policy.

Capability examples:

• can_accept_client_ingress
• can_accept_node_ingress
• can_route_mesh
• can_egress_internet
• can_egress_private_network
• can_run_rdp_worker
• can_run_vnc_worker
• can_run_vpn_exit
• can_run_vpn_connector
• can_run_file_cache
• can_run_update_cache
• can_run_video_relay

Node Role Assignment

node_role_assignments

Purpose: Explicit authorization for a node to execute roles in a cluster and optionally for specific organizations.

Key fields:

• cluster_id
• node_id
• role
• organization_id
• status
• assigned_by_user_id
• assigned_at
• policy

Examples:

• entry-node
• relay-node
• rdp-worker
• vnc-worker
• vpn-exit
• vpn-connector
• file-storage-cache
• update-cache
• video-relay

Node Service Instance

node_service_instances

Purpose: Tracks running service workloads supervised by node-agent.

Key fields:

• cluster_id
• node_id
• service_type
• service_instance_id
• desired_state
• reported_state
• version
• last_heartbeat_at
• metadata

Node Heartbeat

node_heartbeats

Purpose: Durable heartbeat observations and latest state snapshots.

This can be split into an append-only heartbeat log and a compact latest-state table if needed.

Cluster Audit Event

cluster_audit_events

Purpose: Audit cluster, node, enrollment, role, trust, and service-placement changes.

Events should include:

• join token created/revoked
• join request received/approved/rejected
• node certificate issued/rotated/revoked
• role assigned/removed
• service desired state changed
• node health changed
• cluster partition detected/resolved

Future Mesh Entities

These are not Stage C1 implementation requirements, but the data model should not block them:

• mesh_links
• mesh_routes
• cluster_route_intents
• cluster_partitions
• cluster_authority_terms

Node Enrollment Flow

Default enrollment must be explicit and reviewable.

1. Platform admin chooses or creates a cluster.
2. Platform admin creates a short-lived join token scoped to that cluster.
3. Operator installs native rap-node-agent on the host.
4. Node-agent generates a local keypair.
5. Node-agent sends a join request with token, public key, node facts, and capabilities.
6. Control plane validates token hash, expiry, scope, and usage limit.
7. Control plane creates node_join_request in pending state.
8. Platform admin reviews node facts and approves or rejects the request.
9. On approval, control plane creates or activates the node identity.
10. Control plane issues cluster-scoped node certificate or certificate metadata.
11. Node-agent stores identity material locally.
12. Node-agent starts heartbeats.
13. Platform admin assigns roles.
14. Node-agent starts only authorized service workloads.

Auto-approval may exist later only for tightly scoped platform-managed bootstrap tokens. It must not be the default.

Node Runtime Expectations

rap-node-agent should run natively on the host.

The agent owns:

• host identity
• node certificates
• enrollment
• heartbeat
• desired role polling
• service workload supervision
• update trust and rollback
• local recovery

Containerized workloads are preferred for:

• rdp-worker
• vnc-worker
• relay-node
• entry-node
• file-storage-cache
• update-cache
• video-relay

Native workloads are preferred for:

• vpn-exit
• vpn-connector
• host route manager
• firewall/QoS manager
• Windows virtual adapter service
• Android VpnService client

Realtime container workloads should avoid Docker bridge/NAT hot paths where low latency matters. Prefer host networking or native mode for latency-critical paths.

Privileged containers are discouraged. If a workload needs NET_ADMIN, /dev/net/tun, host firewall control, or broad host privileges, native mode should be preferred unless explicitly approved.

Admin Console Model

Admin UI is served through Web Ingress, but cluster configuration belongs to the Control Plane.

The Web/Admin boundary is defined in docs/architecture/WEB_INGRESS_AND_ADMIN_UI_MODEL.md.

Rules:

• Web Ingress is an HTTP/HTTPS entry and presentation layer only
• Web Ingress must not own cluster state, node trust, policy, or secrets
• Admin pages may be dynamically composed from safe ui_manifest / page definitions
• dynamic page definitions must be schema-driven and must not contain internal topology, peer caches, route caches, secrets, raw credentials, or arbitrary executable code
• every cluster mutation still goes through Control Plane authorization, PostgreSQL source-of-truth mutation, and audit
• Fabric Storage / Config Storage role does not imply admin/web ingress role
• adding a storage node to a cluster must not move or create the cluster panel automatically

Admin Endpoint Placement

Admin UI has three distinct scopes:

• Platform Owner Console: global platform-owner scope. It may aggregate visibility across clusters according to platform policy and audit.
• Cluster Admin Endpoint: cluster-local admin/web ingress service for one cluster. It is served only by nodes explicitly assigned an approved admin/web ingress role.
• Organization Admin Panel: tenant-safe projection over allowed organization resources, endpoints, policies, sessions, and safe status.

Storage nodes are configuration distribution/cache nodes. They are not browser admin entry points and they do not own platform or cluster administration.

A cluster-local admin endpoint may become available only after:

• the cluster has healthy authority state
• scoped configuration snapshots are signed and current
• admin/web ingress role is explicitly assigned to authorized node(s)
• TLS/certificate policy is valid
• node health and heartbeat are current
• required role coverage exists for the intended cluster operations

During cluster split/seed workflows, a temporary node may participate in more than one cluster only through isolated memberships, identities, certificates, tokens, storage namespaces, and policies. Removing that seed node from the new cluster is safe only after the new cluster has its own healthy node set, snapshots, role coverage, and admin ingress.

Platform Admin Console MVP

The first admin panel should be for platform owner/admin only.

Required pages:

• Overview
• Clusters
• Nodes
• Join Requests
• Node Detail
• Role Assignments
• Service Workloads
• Health and Alerts
• Audit
• Trust and Certificates

Overview should show:

• cluster count
• node count
• nodes by health
• pending join requests
• role coverage gaps
• service workload health
• risky states

Cluster page should show:

• cluster status
• node membership
• enabled services
• health state
• partition warnings
• role coverage

Node page should show:

• identity
• membership
• health
• capabilities
• assigned roles
• running services
• last heartbeat
• version
• update state
• audit timeline

Join Requests page should allow:

• review node facts
• compare requested capabilities
• approve
• reject
• revoke token

Role Assignments page should allow:

• assign role to node
• restrict role to organization if applicable
• disable role
• audit role changes

Multi-Cluster Admin

Multi-cluster administration should be platform-owner only.

It should support:

• list clusters
• compare cluster health
• inspect node distribution
• inspect service placement
• inspect cluster partitions
• inspect capacity and load
• detect missing role coverage
• manage cluster-scoped trust

Important: No organization should see the full multi-cluster topology.

Multi-cluster trust boundaries:

• platform may operate multiple clusters
• clusters do not automatically trust each other
• clusters do not form one shared mesh by default
• cross-cluster routing requires explicit trust and policy
• a node may participate in multiple clusters only through isolated memberships
• cluster-scoped identities, certificates, tokens, storage namespaces, and policies are required
• platform owner may aggregate visibility across clusters through audited control-plane views
• organization admins see only authorized clusters, resources, services, and safe status

Platform Admin Access Security

Platform owner/admin access must be risk-based.

Trusted or accredited devices may receive lower-friction access, but unknown devices require stronger checks.

Required controls:

• MFA / 2FA policy
• device trust state
• session risk controls
• step-up authentication for high-risk actions
• audit for all high-risk actions

High-risk actions:

• cluster trust changes
• node approval
• role assignment
• partition promotion
• cross-cluster trust
• secrets access
• update policy changes

Future Organization Admin Panel

Organization admin UI should come later.

It should show only safe tenant-scoped objects:

• organization resources
• organization users
• organization policies
• active sessions
• allowed entry points
• allowed egress/service endpoints
• safe VPN/connector status
• audit events for that organization

It must not show:

• full mesh topology
• other organizations
• internal core routing state
• platform-level node trust material
• unrelated cluster internals

Service Visibility Rules

Organizations see only what policy allows:

• resources
• enabled services
• allowed ingress endpoints
• allowed egress/service endpoints
• safe status

Organizations do not see:

• intermediate core mesh nodes
• platform-owned private topology
• other tenants
• cluster-wide trust configuration

Different services may expose different ingress and egress points.

Partition and Split-Brain Behavior

Disconnected minority segments must not:

• approve join requests
• issue node certificates
• assign roles
• change cluster policy
• rotate trust roots

Disconnected segments may continue already-running services only when safe and policy allows it.

Only platform owner authority should promote, merge, split, or recover partitions.

This document does not define final consensus mechanics. It only establishes that cluster authority and partition behavior must be modeled before production mesh changes.

Implementation Stages

Stage C1: Backend Cluster and Node Model Foundation

Status: implemented and verified. Report: artifacts/c1-cluster-node-foundation-report.md.

Goal: Create the explicit backend data model and service boundaries for clusters, join requests, role assignments, node identity, heartbeats, and audit.

Allowed:

• migrations
• repositories
• service interfaces
• platform-admin-only API skeletons
• tests
• docs

Not allowed:

• mesh runtime
• VPN runtime
• node-agent runtime rewrite
• admin UI implementation
• RDP behavior changes

Stage C2: Node Enrollment API

Status: implemented and verified. Report: artifacts/c2-node-enrollment-hardening-report.md.

Goal: Implement production enrollment semantics.

Includes:

• join token creation
• join request creation
• approval/rejection
• certificate metadata boundary
• node activation/revocation
• audit

Stage C3: Native Node-Agent MVP

Status: implemented and verified. Report: artifacts/c3-rap-node-agent-mvp-report.md.

Goal: Create native rap-node-agent MVP.

Includes:

• enroll
• heartbeat
• capability report
• desired role polling
• service status report

No mesh packet routing yet.

Stage C4: Platform Admin Console MVP

Status: implemented and build-verified. Report: artifacts/c4-platform-admin-console-report.md.

Goal: Admin UI for platform operators.

Includes:

• clusters
• nodes
• join requests
• roles
• service health
• audit

Stage C5: Service Workload Supervision Contract

Status: implemented and verified. Report: artifacts/c5-service-workload-supervision-contract-report.md.

Goal: Node-agent can start, stop, and monitor service workloads based on role assignment.

Includes:

• container/native workload contract
• service instance status
• version reporting
• logs/health hooks

Stage C6: Mesh Control-Plane Preparation

Status: implemented and verified. Report: artifacts/c6-mesh-control-plane-preparation-report.md.

Goal: Prepare mesh routing state without carrying production traffic.

Includes:

• link inventory
• route intent
• node reachability
• QoS policy model

Stage C7: Mesh MVP

Status: skeleton implemented and verified. Report: artifacts/c7-mesh-mvp-skeleton-report.md.

Goal: First secure node-to-node overlay path.

Includes:

• mTLS node-to-node
• basic relay skeleton
• route health
• no VPN yet unless separately approved

Stage C8: Multi-Cluster Hardening

Status: implemented and verified. Report: artifacts/c8-multi-cluster-hardening-report.md.

Goal: Safe multi-cluster operations.

Includes:

• cluster authority state
• partition visibility
• cross-cluster admin views
• trust separation

Stage C9: Organization Admin Console

Status: foundation implemented and verified. Report: artifacts/c9-organization-admin-foundation-report.md.

Goal: Tenant-safe admin UI.

Includes:

• resources
• policies
• sessions
• allowed service endpoints
• organization audit

Stage C10: Fabric Core Documentation and Config Distribution Design

Status: completed as documentation/planning stage. Report: artifacts/c10-fabric-core-config-distribution-design-report.md.

Goal: Consolidate Fabric Core terminology and prepare scoped cluster configuration distribution design.

Scope:

• define signed scoped cluster snapshot model boundaries
• define node-local state boundaries
• define peer directory/cache boundaries
• define Fabric Storage / Config Storage role and non-goals
• define PostgreSQL source-of-truth versus distribution/cache boundaries
• define multi-cluster isolation boundaries
• define future implementation stages C11-C19

C10 prepares C11, C12, and C13. It did not implement mesh routing, VPN/IP tunnel runtime, relay packet forwarding, RDP work, code, API changes, or service workload execution.

Stage C11: Signed Scoped Cluster Snapshot Model

Status: completed as documentation/planning stage. Report: artifacts/c11-signed-scoped-cluster-snapshot-model-report.md.

Goal: Define signed, versioned, scoped snapshots for node-local operation and degraded-mode recovery.

Stage C12: Node Local State Store

Status: completed as documentation/planning stage. Report: artifacts/c12-node-local-state-store-report.md.

Goal: Define bounded local state storage for identity, cluster membership, peer cache, route cache, service assignment cache, health, partition/degraded state, config versions, and pending update metadata.

Stage C13: Config / Storage Service Foundation

Status: completed as documentation/planning stage. Report: artifacts/c13-fabric-storage-config-service-report.md.

Goal: Define Fabric Storage Service / Config Storage Service boundaries, replication scope, failure-domain rules, and source-of-truth relationship to PostgreSQL.

Stage C14: Peer Directory and Cache Model

Status: completed as documentation/planning stage. Report: artifacts/c14-peer-directory-cache-model-report.md.

Goal: Define peer directory shape, node-local peer cache, refresh cadence, score-based peer selection inputs, and degraded recovery order.

Stage C15: Fabric Routing Engine Skeleton

Status: completed as documentation/planning stage. Report: artifacts/c15-fabric-routing-engine-skeleton-report.md.

Goal: Introduce a safe compile/runtime boundary for future route selection without carrying production mesh traffic.

Stage C16: Secure Node-to-Node Channel Lifecycle

Status: completed as documentation/planning stage. Report: artifacts/c16-secure-node-to-node-channel-lifecycle-report.md.

Goal: Define authenticated node-to-node channel lifecycle, certificate validation, health, reconnection, and channel authorization.

Stage C17: Mesh Routing Runtime

Status: planning completed. Report: artifacts/c17-mesh-routing-runtime-implementation-plan-report.md. C17A synthetic runtime skeleton is implemented and test-proven. Report: artifacts/c17a-synthetic-mesh-runtime-skeleton-report.md. C17B route health and failover probe skeleton is implemented and test-proven. Report: artifacts/c17b-route-health-failover-probes-report.md. C17C relay semantic hardening skeleton is implemented and test-proven. Report: artifacts/c17c-relay-semantic-hardening-report.md. C17D non-production test-service path experiment is implemented and test-proven. Report: artifacts/c17d-non-production-test-service-path-report.md. C17E live node-to-node synthetic HTTP transport skeleton is implemented and smoke-proven. Report: artifacts/c17e-live-node-to-node-synthetic-transport-report.md. C17F scoped synthetic peer/route config loading and route-health reporting is implemented and smoke-proven. Report: artifacts/c17f-scoped-synthetic-route-config-report.md. C17G Control Plane scoped synthetic config read boundary is implemented and test-proven. Report: artifacts/c17g-control-plane-scoped-synthetic-config-report.md.

Goal: Define the mesh routing runtime implementation plan only after identity, enrollment, scoped config, local state, storage, peer cache, routing skeleton, and secure node-to-node channel lifecycle stages are accepted.

First implementation step was C17A: synthetic fabric messages only, feature-flagged, test topology only, no RDP/VPN/service traffic, and no topology exposure to organizations.

C17D proved a bounded synthetic.echo test-service path over direct, single-relay, and forced fallback routes. It does not authorize production service traffic.

C17E proved the same synthetic-only route model over real local HTTP node endpoints using a disabled-by-default rap-node-agent endpoint and a mesh-live-smoke harness. It still does not authorize production mesh traffic, RDP/VPN traffic, service workload traffic, or organization-visible topology.

C17F moved the C17E synthetic route model from manual debug JSON toward scoped configuration by adding a node-local scoped config file boundary and synthetic route-health reporting to the Control Plane mesh link endpoint. It still does not authorize production mesh traffic.

C17G added the Control Plane read endpoint for node-scoped synthetic mesh config and node-agent consumption of that config. It still does not authorize production mesh traffic.

Stage C18: VPN / IP Tunnel Service

Status: completed as documentation/planning stage. Report: artifacts/c18-vpn-ip-tunnel-service-target-design-report.md.

Goal: Define VPN/IP tunnel service as a cluster-managed service above Fabric Core and Fabric Routing, not as node-local ad hoc configuration.

Result:

• target design: docs/architecture/VPN_IP_TUNNEL_SERVICE_TARGET.md
• defines vpn_connection as logical desired state
• defines single-active lease/fencing model
• defines control-plane versus data-plane boundary
• defines routing policy, QoS, security, credential distribution, audit, and failure behavior
• defines future C18A-C18I stages

C18 did not implement VPN/IP tunnel runtime, TUN/TAP devices, packet routing, mesh production traffic, RDP work, API changes, migrations, or service workload execution.

Stage C18A: VPN / IP Tunnel Control-Plane Data Model

Status: implemented and backend-test-proven. Report: artifacts/c18a-vpn-control-plane-data-model-report.md.

Goal: Add the durable control-plane source-of-truth model for future VPN/IP tunnel service desired state without implementing VPN runtime.

Completed:

• vpn_connections
• vpn_connection_allowed_nodes
• vpn_connection_route_policies
• vpn_connection_leases
• backend models, repository methods, service methods, and platform-admin API skeleton
• single-active lease boundary with PostgreSQL unique active lease protection
• recovery-admin-only lease fencing boundary
• tests for authorization, scope, desired-state gating, duplicate active lease rejection, route policy validation, and allowed-node normalization

C18A did not implement VPN/IP tunnel runtime, TUN/TAP devices, packet routing, host route/firewall manipulation, node-agent execution, production mesh traffic, RDP work, Windows client changes, or data-plane behavior changes.

Stage C18B: VPN / IP Tunnel Lease and Fencing Hardening

Status: implemented and backend-test-proven. Report: artifacts/c18b-vpn-lease-fencing-hardening-report.md.

Goal: Harden the future VPN/IP tunnel single-active control-plane ownership model before any node-agent executes VPN work.

Completed:

• owner eligibility validation against active cluster membership
• owner validation against vpn_connection allowed-node policy
• owner validation against active vpn-exit or vpn-connector role assignment
• same-owner acquire idempotency
• different-owner active lease rejection
• idempotent release/fence behavior
• stale lease cleanup/reclamation service boundary
• audit events for lease renewal and stale lease expiration
• tests for wrong cluster, unauthorized owner, missing role, disabled connection, expired lease renewal, duplicate active lease, and stale lease audit behavior

C18B did not implement VPN/IP tunnel runtime, TUN/TAP devices, host route/firewall manipulation, node-agent VPN execution, production mesh traffic, RDP work, Windows client changes, or data-plane behavior changes.

Stage C18C: VPN / IP Tunnel Node-Agent Desired-State Consumption

Status: implemented and backend-test-proven. Report: artifacts/c18c-vpn-node-agent-desired-state-report.md.

Goal: Expose scoped VPN/IP tunnel desired assignments to eligible node-agents and allow node-agents to report observed assignment status without executing VPN runtime.

Completed:

• node-agent desired-state API for scoped VPN assignments
• node-agent assignment status report API
• explicit observed status values: not_started, assigned, lease_required, blocked, unknown
• PostgreSQL assignment status report and latest-status tables
• backend service/repository boundaries for scoped assignment visibility
• assignment visibility limited to eligible candidates or current active owner
• credential_ref withheld from node-agent assignment payloads; only has_credential_ref is exposed for future resolver integration
• tests for unauthorized/invisible assignment rejection, allowed status values, invalid status rejection, and non-platform-admin node-agent read path

C18C did not implement VPN/IP tunnel runtime, TUN/TAP devices, host route/firewall/DNS/QoS manipulation, credential delivery, node-agent VPN execution, production mesh traffic, RDP work, Windows client changes, or data-plane behavior changes.

Stage C19: Version Storage and Node Update / Rollback Foundation

Status: future stage. Architecture direction is documented in this file and docs/architecture/SECURE_ACCESS_FABRIC_TARGET.md. No updater runtime is implemented by this documentation.

Goal: Define the durable release, artifact, node-agent update, health supervision, rollback, and data-structure migration model before any production update runtime is enabled.

Scope:

• define version-storage / update-repository as the signed release artifact source for node-agent and service workloads
• define release channels: stable, current, and candidate
• define OS / architecture artifact variants under a single signed release manifest
• define last-known-good stable release tracking and rollback eligibility
• define update-cache mirroring of approved artifacts without becoming a source of truth
• define node-agent download, verification, staging, restart, health-check, promote, rollback, and failure-reporting states
• define migration bundle metadata for PostgreSQL schema, service-local data, node-local state, config/cache shards, and protocol/config schema changes
• define rollout policy boundaries for canary, staged rollout, pause, resume, rollback, blocklist, and emergency pinning

Non-goals:

• no production updater runtime
• no automatic schema migration execution by node-agent for PostgreSQL
• no unsigned or unapproved artifacts
• no arbitrary code download
• no mesh, VPN/IP tunnel, RDP, client, or data-plane behavior changes

Rules:

• PostgreSQL remains authoritative for rollout policy, approvals, desired versions, channel assignment, migration approval, and audit.
• Version Storage stores immutable signed artifacts, release manifests, compatibility metadata, hashes, signatures, provenance, and migration bundles.
• Update-cache nodes only mirror approved artifacts and may be invalidated or rebuilt from Version Storage.
• Node-agent executes only signed, approved, scoped work assigned by policy.
• Node-agent must stop, restart, rollback, or fence local workloads according to health checks and lease/policy state.
• Node-agent self-update requires stricter safeguards than workload updates: staged replacement, watchdog, crash-safe rollback, and preservation of control-plane/update-source reachability unless an approved break-glass procedure exists.
• PostgreSQL schema migrations are orchestrated by the Control Plane release process, not independently invented by a node.
• Service-local and node-local migrations may be executed by node-agent only when declared in a signed release manifest and scoped to that node/workload.

Historical Stage C1 Prompt

This prompt is retained for provenance only. Stages C1-C9 are now listed above as implemented/verified or build-verified according to their reports.

Proceed with Stage C1 only.

Goal: Implement backend cluster and node model foundation for the Secure Access Fabric.

Strict rules:

• do NOT implement mesh runtime
• do NOT implement VPN runtime
• do NOT change RDP runtime behavior
• do NOT build admin UI yet
• do NOT rewrite existing node-agent runtime
• do NOT expose full topology to organizations
• keep PostgreSQL as source of truth
• keep Redis for live coordination only

Scope:

1. Add explicit cluster model.
2. Add cluster membership model.
3. Add node join token model with hashed tokens only.
4. Add node join request model.
5. Add node identity/certificate metadata model.
6. Add node role assignment model.
7. Add node heartbeat/latest health model.
8. Add cluster audit events.
9. Preserve existing node tables with safe migration/backfill into a default cluster.
10. Add repository interfaces and PostgreSQL implementations.
11. Add service/usecase boundaries for platform-admin cluster and node management.
12. Add platform-admin-only API skeletons for clusters, nodes, join requests, and role assignments.
13. Add tests for cluster scoping, role authorization, join token hashing, and organization topology isolation.
14. Update documentation.

Deliver:

• migrations
• backend models/repositories/services
• platform-admin API skeleton
• tests
• docs update
• verification report

Historical note: C2 was not to start until C1 was accepted. C1-C9 are now recorded above with their current accepted statuses.

Result / Decision

This document defines Fabric Core as the foundation beneath service adapters and before mesh runtime. It clarifies peer discovery, score-based routing, scoped configuration distribution, node-local state, Fabric Storage/Config Storage, multi-cluster trust boundaries, and risk-based platform admin access. It also hardens the boundaries that Redis is live-only, Fabric routing must not depend on live backend availability, and Fabric Storage/Config Storage is not a general-purpose distributed database or second source of truth.

Stage C10 through Stage C18 planning are completed as documentation/planning. C17A, C17B, C17C, C17D, C17E, C17F, and C17G are implemented and test-proven with synthetic traffic only. C18 defines the VPN/IP tunnel service target model but does not authorize VPN/IP tunnel runtime. C18A adds the VPN/IP tunnel control-plane data model and platform-admin skeleton only. C18B hardens single-active lease/fencing semantics. C18C adds node-agent desired-state/status reporting for scoped VPN assignments only. C19 is now reserved for the Version Storage/Update Repository and node-agent update/rollback foundation; it is not implemented by this document. No RDP, data-plane, VPN runtime, production relay, production mesh service traffic, node-agent VPN execution, host networking, service workload runtime, or production updater behavior is implied by this document.