Files

T

m 469fa0e860 3

2026-05-18 21:33:39 +03:00

44 KiB

Raw Blame History

Data Plane v1 for RDP

Archived status: this document is a historical RDP/WebSocket stage record, not the current runtime source of truth for transport architecture. The active fabric transport model is QUIC-only between nodes; see docs/architecture/DISTRIBUTED_FABRIC_NODE_PROTOCOL_PLAN.md, docs/architecture/FABRIC_FIRST_TRANSPORT_AND_STRESS_PLAN.md, and docs/architecture/SECURE_ACCESS_FABRIC_TARGET.md.

Status: DP-3A grayscale full-frame binary render foundation is implemented and smoke-proven on the test Docker environment as of 2026-04-25. DP-3B adaptive quality policy/selection is intentionally paused. The accepted C++ RDP Adapter baseline is the ordered-region path. RDP-Perf-6 makes direct dirty-region binary render explicit with render.frame.full / render.frame.region RAP2 message types and is build/probe/live-smoke-proven on the test Docker environment as of 2026-04-26. The current test Docker deployment for the RDP Adapter performance path is rap-rdp-worker:rdp-perf6-dirty-region. The Stage 5.2 core download data path remains runtime-proven for direct worker WSS and backend gateway fallback. Data-plane and RDP work are paused; the next active focus is Stage C10 Fabric Core / cluster foundation, not another data-plane feature.

This document defines the first staged data-plane evolution for the RDP MVP. It does not implement direct worker WebSocket runtime, mesh routing, VPN, QUIC, UDP, WebRTC, relay nodes, or multi-cluster behavior.

The long-term platform target is defined in docs/architecture/SECURE_ACCESS_FABRIC_TARGET.md. This document narrows that target to DP-1: direct client-to-worker WSS for RDP realtime traffic, with the current backend gateway retained as fallback/debug.

1. Current Problem

The current RDP MVP routes realtime input/render through the backend WebSocket gateway and Redis-backed coordination. This is acceptable for fallback, debugging, lifecycle proof, and early MVP validation.

It is not acceptable as the production realtime path because:

render frames are high-rate and high-volume
base64/JSON render payloads add CPU and payload overhead
backend gateway can become a bottleneck under concurrent sessions
input can compete with render/frame processing
backend API capacity should be reserved for control-plane work
Redis must not become a frame transport or durable render store

The current implementation remains valid as fallback while DP-1 is introduced in stages.

2. Target DP-1 Path

Target DP-1 path:

Windows client
  -> direct WSS data-plane connection
  -> rdp-worker realtime endpoint
  -> existing RDP session runtime
  -> FreeRDP

Control-plane path remains:

Windows client
  -> backend API
  -> auth / org / policy / session broker
  -> worker selection
  -> short-lived data-plane token issuance

Fallback path remains:

Windows client
  -> backend WebSocket gateway
  -> current gateway/Redis/worker coordination path

DP-1 does not replace the session broker. It only moves realtime session traffic away from backend relay when direct worker WSS is available and authorized.

3. Responsibilities

Backend as Control Plane

Backend remains responsible for:

authentication
organization selection and isolation
resource authorization
resource policy evaluation
session lifecycle
worker selection
attachment ownership
takeover semantics
audit
short-lived data-plane token issuing
returning data-plane candidates
retaining backend gateway fallback

Backend must not become the production high-rate render relay.

Worker as Direct Realtime Endpoint

The RDP worker becomes responsible for:

exposing an authorized direct WSS endpoint
validating data_plane_token
binding a WSS connection to an existing session runtime
enforcing session, attachment, user, organization, and channel scope
carrying realtime logical channels directly:
- input
- render
- clipboard
- file_upload
- control / heartbeat
- telemetry
preserving existing FreeRDP runtime boundaries
preserving policy enforcement already present in worker runtime

The worker must not create a new RDP session just because a direct WSS connection attaches. It must bind to the existing broker-created session runtime.

Windows Client

The Windows client will eventually:

read data-plane candidates from session start/attach responses
prefer direct_worker_wss when available
fall back to backend_gateway when direct worker WSS is unavailable
keep existing lifecycle behavior unchanged
keep backend gateway support for debug/fallback

No client behavior changes are required for DP-1A.

4. Backend Contract Proposal

On session start, attach, and takeover, backend should extend the response with data-plane candidates.

Example:

{
  "session_id": "session-123",
  "attachment_id": "attachment-456",
  "gateway_url": "wss://backend.example.com/api/v1/gateway/ws",
  "data_plane": {
    "preferred": "direct_worker_wss",
    "token": "short-lived-data-plane-token",
    "expires_at": "2026-04-25T13:00:00Z",
    "candidates": [
      {
        "type": "direct_worker_wss",
        "url": "wss://worker-node.example.com/rap/v1/data-plane"
      },
      {
        "type": "backend_gateway",
        "url": "wss://backend.example.com/api/v1/gateway/ws"
      }
    ]
  }
}

Compatibility rules:

Existing fields must remain valid.
Existing clients may ignore data_plane.
gateway_url remains available for fallback/debug.
The backend must not return direct worker candidates unless the worker is live and route policy permits it.
Token TTL must be short.

Proposed DTO Shape

Names are proposals only.

SessionControlResult
  session
  attachment
  attach_token
  gateway_url
  data_plane?

DataPlaneOffer
  preferred
  token
  expires_at
  candidates[]

DataPlaneCandidate
  type
  url
  worker_id?
  node_id?
  cluster_id?
  priority?
  metadata?

5. Data Plane Token Model

data_plane_token must be short-lived and scoped. It is not a general API token.

Required claims:

session_id
attachment_id
user_id
organization_id
cluster_id if available
worker_id
resource_id
allowed_channels
expires_at
nonce / jti
issued_at
issuer
audience

Allowed channel values:

input
render
clipboard
file_upload
control
telemetry

Validation rules:

token must be signed by the backend with RS256 private key
worker must validate with public key only and must not hold a signing secret
token must be short-lived
token must match the worker receiving it
token must match an active session runtime
token must match current attachment/controller where required
token must not grant channels not allowed by resource policy
token must not survive session termination
token replay must be rejected or bounded by jti / nonce cache

Token refresh is not part of DP-1A. Future stages may either reissue tokens through the control plane or renew direct connections through a controlled flow.

6. Direct WSS Channel Model

DP-1 uses a single WSS connection with logical channels. Later stages may split transports, but DP-1 must keep the model simple and bounded.

`control`

Reliable channel.

Used for:

attach handshake
heartbeat
session state messages
detach notification
takeover notification
terminate notification
protocol errors

`input`

Highest-priority channel.

Rules:

input never waits behind render
key down/up and mouse button/wheel events must be ordered
mouse move may be coalesced to latest
input queues must be bounded
stale mouse move may be dropped
click/key/wheel must not be dropped under normal operation

`render`

Droppable/latest-frame channel.

Rules:

stale render frames must be dropped
latest frame wins
render must not block input/control
binary payloads should be used on direct data plane
backend fallback may continue existing JSON/base64 behavior during migration

`clipboard`

Reliable policy-gated channel.

Rules:

existing clipboard_mode applies
text-only behavior remains until richer formats are explicitly designed
blocked behavior must remain localized in clients
worker must enforce policy again

`file_upload`

Reliable chunked channel.

Rules:

existing file_transfer_mode applies
bounded chunk size
content hash
transfer id
no arbitrary path exposure
file upload must not block input

`telemetry`

Low-priority channel.

Rules:

sampled or lossy telemetry is acceptable
telemetry must not block user traffic
useful metrics include input RTT, frame FPS, dropped frames, queue length, decode time, render apply time

7. Message Framing

DP-1 uses:

JSON control messages for small envelopes
binary WebSocket frames for render payloads
no base64 for direct data-plane render frames

Backend fallback keeps the current JSON/base64 frame path for debug/fallback. Direct worker WSS uses binary render frames when the backend advertises render_transport=binary_v1 and the client requests render_transport=binary_v1.

JSON Envelope

Small control/reliable messages may use JSON:

{
  "protocol_version": 1,
  "session_id": "session-123",
  "attachment_id": "attachment-456",
  "channel": "input",
  "message_type": "mouse",
  "sequence": 1024,
  "timestamp": "2026-04-25T13:00:00.000Z",
  "flags": {},
  "payload": {}
}

Binary Frame Header

DP-2 uses a fixed 16-byte preamble followed by a UTF-8 JSON header and a raw binary payload:

offset  size  field
0       4     magic = "RAP2"
4       2     protocol_version, little-endian uint16, currently 1
6       2     flags, little-endian uint16
8       4     header_length, little-endian uint32
12      4     payload_length, little-endian uint32
16      n     UTF-8 JSON header
16+n    m     raw render payload bytes

The DP-2 JSON header contains:

protocol_version
session_id
channel, currently render
message_type, currently render.frame.full or render.frame.region on direct worker WSS; session.frame remains accepted as the legacy DP-2 binary message type for compatibility.
sequence
timestamp
flags
payload_length
frame_width
frame_height
frame_stride
frame_format
optional region fields when message_type=render.frame.region: region_x, region_y, region_width, region_height, region_stride, region_format=BGRA32
optional color_mode, currently full_color or grayscale
optional quality_profile
optional original_frame_format
optional output_frame_format
optional raw_frame_bytes
optional binary_direct_bytes
optional diagnostics: full_frame_bytes, region_bytes, region_savings_percent, diff_time_ms, render_update_reason, fallback_to_full_frame_reason
optional input_correlation_id
optional worker_frame_captured_at

Binary frames must include a fixed or clearly parseable header before payload.

Required header fields:

protocol_version
session_id
channel
message_type
sequence
timestamp
flags
payload_length

Render payload must not be base64 encoded on direct data plane.

Suggested render message types:

render.frame.full
render.frame.region
render.cursor
render.resize
render.quality.changed

Suggested flags:

keyframe
droppable
latest_only
compressed
interactive
grayscale

8. Quality Profile Foundation

DP-1A defines quality profiles only. It does not implement adaptive rendering.

Profiles:

emergency_grayscale
low_bandwidth
text_priority
balanced
high_quality

Color modes:

full_color
256_colors
64_colors
16_colors
grayscale

Rules:

quality profile must affect real render behavior in later stages
input priority remains absolute
render quality must degrade before input latency increases
lower profiles may reduce FPS, color depth, region size, or compression settings
higher profiles may increase FPS and color fidelity only when queues remain healthy
profile selection must be policy-aware and observable

9. Security Model

DP-1 security boundaries:

backend authorizes session access
backend issues short-lived data-plane token
worker validates token before accepting direct WSS
worker binds token to existing session runtime
worker enforces channel permissions
worker rejects mismatched session, attachment, organization, resource, worker, or expired token
backend gateway fallback keeps existing auth path

Transport:

direct worker WSS must use TLS
future node-to-node traffic uses mTLS as defined in the Secure Access Fabric target
DP-1 direct WSS may start with worker server TLS plus signed token validation
P3.2 direct worker WSS trust metadata distinguishes smoke_insecure, public_ca, and platform_ca
production backend must not advertise smoke-only direct candidates
production clients must not use insecure TLS bypass and must fall back to backend gateway if direct worker trust is unavailable
production deployments should avoid long-lived static worker secrets

Audit:

backend audits token issuance
backend audits session lifecycle
worker should report direct attach/detach/failure events back to control plane
direct data-plane traffic does not require auditing every input/render event
high-risk events such as takeover, failed token validation, policy denial, and file transfer should be auditable

10. Fallback Backend Gateway Path

The current backend WebSocket gateway remains:

fallback path
debug path
compatibility path for older clients
smoke-test path while DP-1 is staged

Fallback activation cases:

no direct worker candidate returned
direct WSS connect fails
token validation fails due to stale route
worker endpoint unavailable
policy forces backend gateway
client version does not support direct WSS

Fallback rules:

fallback must preserve existing lifecycle behavior
fallback must not silently weaken policy
fallback should be visible in logs/telemetry
fallback should be measurable against direct path latency

11. Migration Stages

Stage DP-1A: Spec Only

Create architecture/spec documentation.

No runtime behavior changes.

Stage DP-1B: Backend Offers Data Plane Candidates

Status: completed.

Backend extends session start/attach/takeover responses with data_plane.

Client still uses fallback backend gateway.

Implementation status:

backend response DTO can include optional gateway_url and data_plane
data_plane.token is a short-lived signed token with session, attachment, user, organization, worker, resource, allowed-channel, expiry, and jti scope
backend_gateway candidate is always returned when configured
direct_worker_wss candidate is returned only when a direct worker WSS URL template is configured
current clients may ignore data_plane safely
no worker direct WSS runtime is implemented in this stage
no client routing behavior changes in this stage

Verification:

old clients still work
responses include valid candidate shape
token is short-lived
token is scoped

Stage DP-1C: Worker Direct WSS Endpoint

Status: completed.

Worker exposes direct WSS endpoint and validates data_plane_token.

Windows client still uses fallback backend gateway.

Implementation status:

worker has optional /rap/v1/data-plane WSS endpoint
endpoint is disabled by default and requires TLS certificate/key paths
worker validates signed RS256 data_plane_token with a public key only
worker keeps no data-plane signing secret
worker rejects reused jti values with a bounded in-memory TTL cache
token validation checks session, attachment, user, organization, worker, resource, allowed channels, expiry, audience, and jti
endpoint binds only to existing SessionRuntime
bind checks reject old attachment after takeover, wrong attachment, wrong worker, wrong organization, wrong resource, missing runtime, failed/terminated runtime state, and channels broader than runtime policy
invalid token, wrong worker, expired token, replayed jti, and missing runtime are rejected
rdp-worker-dataplane-token-probe validates token behavior in the worker image
rdp-worker-dataplane-bind-probe validates attachment/state/channel bind policy without starting RDP
backend gateway remains active fallback
no Windows client routing change is included in this stage

Verification:

token validation works
runtime binding rejects missing runtime without creating a new RDP session
replayed jti values are rejected
wrong attachment and over-broad channels are rejected
no new RDP session is created
invalid tokens are rejected

Stage DP-1D: Windows Client Prefers Direct WSS

Status: completed as hardened client transport selection.

Windows client uses direct worker WSS only when the candidate is explicitly marked data-capable. Current DP-1C worker endpoint validates and binds but does not yet carry production render/input traffic, so unmarked candidates fall back to the backend gateway immediately.

Implementation status:

Windows session DTOs understand optional data_plane offers and candidates
transport selection remains behind ISessionGatewayClient
direct worker WSS candidates are considered only when metadata contains runtime_transport=json_v1 or traffic_ready=true
direct WSS attach attempts use short bounded timeout and never block the UI
failed/unavailable/not-ready direct path automatically uses backend gateway
existing backend gateway behavior remains unchanged
no worker runtime changes are included in DP-1D
no binary render frames or mesh/relay/VPN behavior is included

Verification:

Windows client build succeeds
fallback works and remains the default runtime path for current DP-1C endpoint
direct candidate selection is capability-gated to avoid losing render/input
lifecycle behavior remains stable

Stage DP-1D.1: Worker Direct JSON Realtime Bridge

Status: runtime-proven on the test Docker environment as of 2026-04-25.

Worker direct WSS now carries the same JSON realtime envelopes already used by the backend gateway. This is intentionally a bridge stage, not the final production data-plane protocol.

Implementation status:

worker direct WSS accepts existing JSON input, control, clipboard, and file_upload envelopes
worker direct WSS emits existing JSON session.state, session.frame, session.taken_over, clipboard.text, and file_upload.progress events
direct WSS binds only to an existing SessionRuntime; it never creates a new RDP runtime
direct inbound envelopes are bounded and drained before Redis fallback input
mouse move can be coalesced, but click, wheel, keyboard, clipboard, and file upload envelopes remain reliable within bounded queues
direct render is latest-frame-only and droppable in the worker WSS writer
direct inbound envelopes are tagged with token-bound session, attachment, user, organization, worker, and resource claims before they enter runtime
runtime rejects direct envelopes whose attachment_id no longer matches the current active controller attachment
takeover updates emit session.taken_over to the previous direct attachment while normal frame/state events continue only to the current attachment
backend advertises direct metadata only when DATA_PLANE_DIRECT_WORKER_JSON_RUNTIME=true
backend gateway fallback remains active and unchanged
Windows client behavior remains gated by DP-1D metadata selection

Verification performed:

backend go test ./... passes
Windows client build passes with no routing behavior change required
worker canonical Docker image builds with the direct JSON bridge
DP-1C endpoint smoke still proves malformed token rejection, valid-token-without-runtime rejection, and jti replay rejection
backend tests prove runtime_transport=json_v1 and traffic_ready=true are emitted only when the explicit runtime flag is enabled
live runtime proof was run on test Docker 192.168.200.61 with DATA_PLANE_DIRECT_WORKER_JSON_RUNTIME=true
backend session start returns direct_worker_wss candidate metadata: runtime_transport=json_v1 and traffic_ready=true
Windows desktop smoke selected direct_worker_wss and connected to wss://192.168.200.61:18443/rap/v1/data-plane
worker direct WSS validated the token and bound to the existing runtime
direct WSS accepted input envelopes and applied mouse/keyboard events through FreeRDP
direct WSS emitted JSON render/state events and the Windows client rendered a real desktop frame
direct WSS carried text clipboard client-to-server through the existing clipboard envelope and worker policy/cliprdr boundary
direct WSS carried chunked file upload through existing file_upload.start / file_upload.chunk envelopes and emitted file_upload.progress
fallback was proven by advertising an unavailable direct worker URL; the Windows client timed out direct WSS and selected backend_gateway
detach, reattach, takeover, session.taken_over, input, and render remained stable in direct and fallback smoke runs
no new RDP runtime was created by direct WSS attach; worker logs showed one started new runtime for the session and later updated assignment for existing session on reattach/takeover

Known limitations after DP-1D.1:

direct render still uses JSON/base64 full-frame payloads; binary render frames remain DP-2
direct server-to-client clipboard was not re-matrixed in this DP proof because Stage 4.1 already proved FreeRDP cliprdr behavior; DP-1D.1 proved that the direct bridge carries clipboard envelopes and preserves worker enforcement
file upload direct proof lands in the existing restricted worker visible transfer directory; broader file-transfer UX remains outside DP-1D.1
the Windows smoke script reports rendering=false when compact layout hides telemetry controls, even though frame receipt/rendering is proven by logs and UIA event text

Stage DP-1E: Latency Comparison

Status: measurement-complete on the test Docker environment as of 2026-04-25.

Compare direct path vs fallback before starting DP-2 binary render frames.

Metrics:

input capture to worker apply
worker frame capture to client render
frame queue length
dropped stale frames
close/dispose latency
fallback activation count

Smoke commands used:

pwsh -ExecutionPolicy Bypass -File scripts/windows-smoke/desktop-smoke.ps1 `
  -PreferDirectDataPlane:$true `
  -AllowInsecureDirectDataPlaneTlsForSmoke:$true `
  -DirectDataPlaneConnectTimeoutMs 2500 `
  -SkipOrgSwitchAndTokenRefresh

pwsh -ExecutionPolicy Bypass -File scripts/windows-smoke/desktop-smoke.ps1 `
  -PreferDirectDataPlane:$false `
  -AllowInsecureDirectDataPlaneTlsForSmoke:$true `
  -DirectDataPlaneConnectTimeoutMs 750 `
  -SkipOrgSwitchAndTokenRefresh

Measured sessions:

direct worker WSS: 59af4b37-3708-4cff-8e9d-054869946250
backend gateway fallback baseline: 673b7540-6276-4d73-824b-e5b2ea96182a
additional fallback-activation proof: direct candidate unavailable/not-ready logs on 8d89dd5c-fb14-4f70-a4e4-01ebb2a37da4 and 673b7540-6276-4d73-824b-e5b2ea96182a

Verification summary:

direct smoke passed login, resource list, start, input, detach, reattach, takeover, session.taken_over, and logout
fallback smoke passed login, resource list, start, input, detach, reattach, takeover, session.taken_over, and logout
smoke rendering=false is a compact-layout harness artifact; session event log contained Desktop frame received and client logs contained SessionWindow rendered frame
cleanup probe against /api/v1/sessions/active returned 404 because that endpoint does not exist; the implemented list endpoint is /api/v1/sessions?user_id=...
Redis worker queues were empty after the measured runs: worker:queue:59af... = 0, worker:queue:673b... = 0

Latency matrix:

Metric	Direct worker WSS	Backend gateway fallback
Client transport selection	`selected=direct_worker_wss` in desktop logs	`selected=backend_gateway` in desktop logs
Client capture/send to worker apply	direct smoke retained worker-side receive/apply timestamps; client capture timestamp was not retained in the compact smoke log	sampled fallback activation: about `205ms` from WPF capture to worker apply for mouse down
Backend gateway input hop	bypassed for direct realtime input	backend receive to route typically `<1ms`
Worker receive to FreeRDP apply, mouse down	`0ms` to `24ms` observed	`0ms` to `29ms` observed
Worker receive to FreeRDP apply, mouse up	`25ms` to `26ms` observed	`28ms` to `29ms` observed
Worker receive to FreeRDP apply, key down	about `25ms` observed	about `33ms` observed
Worker receive to FreeRDP apply, key up	about `26ms` observed	about `30ms` observed
Backend route to worker receive	not applicable	about `31ms` for key down, about `0ms` to `31ms` for sampled mouse/key events
FreeRDP apply to next captured frame	`0ms` to `40ms` observed	`0ms` to `43ms` observed
Worker frame capture to backend receive	backend still receives worker frame telemetry; observed same-second receive	same-second receive
Backend frame receive to client write	not on direct render path	sampled `486ms` and `753ms` on full-frame JSON/base64 gateway writes
Client render proof	`session.frame` received and frame rendered in direct smoke	`SessionWindow rendered frame seq=19 size=1280x720`
SessionGatewayConnection dispose	about `1ms` in sampled close traces	about `1ms` in sampled close traces
SessionWindow closed handler	below `1ms` in sampled close traces	below `1ms` in sampled close traces

Queue and backpressure observations:

direct inbound drained bounded batches before fallback Redis input
direct mouse move coalescing was active while preserving click/key ordering
direct outbound reported frames_queued_per_second matching frames_sent_per_second; reliable_dropped=0
worker render pending remained 0 for both paths
fallback Redis append queue length stayed bounded in sampled logs, usually 1 to 3, and returned to 0 after the run

Render observations:

direct render is already latest-frame-only/droppable at the worker WSS writer
worker render rates during interaction were approximately:
- direct: ~3.0 to ~5.7 frames/sec sent/published, pending 0
- fallback: ~2.0 to ~5.0 frames/sec published, pending 0
current frames are still JSON/base64 full-frame payloads
measured frame payload size remains about 4,915,200 bytes per JSON/base64 frame, so DP-1D.1 improves routing but does not remove the render payload bottleneck

Fallback activation proof:

fallback was explicitly selected when the client was configured with PreferDirectDataPlane=false
fallback was also visible when direct WSS was unavailable or not runtime ready, with client logs:
- data_plane.transport direct_worker_wss failed; falling back to backend_gateway
- data_plane.transport direct_worker_wss unavailable_or_not_runtime_ready; using backend_gateway
- data_plane.transport selected=backend_gateway

DP-1E conclusion:

direct worker WSS removes backend/Redis from the realtime input path
fallback backend gateway remains functional and observable
neither path showed unbounded input queue growth during smoke
close/dispose traces remained fast in sampled logs
the dominant remaining bottleneck is render payload format and size, not worker input scheduling
DP-2 should focus on binary render frames and avoiding base64/JSON render payloads on the direct data plane

Stage DP-2: Binary Render Frames

Status: implemented and smoke-proven on the test Docker environment as of 2026-04-25.

Direct worker WSS now sends render payloads as binary WebSocket frames when the backend candidate metadata advertises render_transport=binary_v1 and the Windows client requests that transport. Backend gateway fallback continues to use the existing JSON/base64 frame path.

Goals:

remove base64 overhead from the direct worker WSS wire path
reduce direct render payload size
keep backend gateway JSON/base64 fallback intact
keep direct render latest-frame-only and droppable
keep input/control ahead of render

Implementation notes:

Backend advertises binary direct render only when DATA_PLANE_DIRECT_WORKER_BINARY_RENDER=true.
Direct candidate metadata includes runtime_transport=json_v1, traffic_ready=true, and render_transport=binary_v1.
Worker direct WSS accepts existing JSON envelopes for control/input/clipboard/file_upload and emits binary WebSocket frames for session.frame.
Windows client enables binary parsing only for direct candidates that advertise render_transport=binary_v1 or binary_render=true.
Backend gateway fallback remains unchanged and continues to deliver session.frame as JSON/base64.

Smoke proof:

direct session id: 824c0057-c8a0-4366-b5c2-805597ae2d61
fallback session id: 28e4b198-2c27-4971-951a-7b187c11f96d
direct client selected direct_worker_wss with render_transport=binary_v1
direct worker bind succeeded with render_transport=binary_v1
client received binary frames with raw payload size 3,686,400 bytes
client rendered binary frames, including frame sequences 1, 2, 4, 7, 9, 12, 14, 15, 17, 18, and 19
fallback client selected backend_gateway
fallback rendered JSON/base64 frames through the existing backend gateway path

Payload comparison:

DP-1E JSON/base64 frame payload: about 4,915,200 bytes for 1280x720 BGRA
DP-2 direct binary frame payload: 3,686,400 bytes for the same 1280x720 BGRA frame, plus a small binary preamble and JSON header
Direct wire payload reduction is about 25 percent compared with base64.

Latency and queue observations from smoke:

direct click frame render sample: worker captured frame at 1777141091937, WPF rendered it at 2026-04-25T21:18:11.6628382+03:00, about 226 ms later
direct key-down frame render sample: worker captured frame at 1777141093434, WPF rendered it at 2026-04-25T21:18:13.1614990+03:00, about 727 ms later
direct worker render rate sample: seen_per_second=4.953283, published_per_second=3.962626, dropped_per_second=0.990657, pending=0
direct data-plane outbound sample: frames_queued_per_second=5.404927, frames_sent_per_second=5.404927, binary_render_bytes_per_second=19926577.806299, json_render_bytes_per_second=0.000000, reliable_dropped=0
fallback worker render rate sample: seen_per_second=4.871576, published_per_second=3.897260, dropped_per_second=0.974315, pending=0

Known limitations:

DP-2.1 removed the internal base64 encode/decode hop from the direct render path. The direct worker WSS sender now receives raw captured frame bytes and writes them into RAP2 binary frames without decoding a compatibility session_frame.
The worker still builds compatibility session_frame events with base64 for backend gateway/live-state fallback. That compatibility conversion is intentionally isolated to the fallback boundary and is not used by the direct binary render sink.
Backend still receives compatibility worker frame events for fallback/debug. Binary render frames are not routed through Redis or backend gateway.
At the DP-2.1 point, dirty regions, tile encoding, adaptive quality, compression/codecs, and color-mode reduction remained later work.
Smoke rendering=false remains a compact-layout harness artifact; UIA output and client logs prove Desktop frame received and SessionWindow rendered frame.

Stage DP-2.1: Worker Raw-Frame Split

Status: implemented and smoke-proven on the test Docker environment as of 2026-04-25.

DP-2.1 keeps the DP-2 RAP2 binary frame contract and removes the remaining worker-internal base64 encode/decode hop from the direct render path.

Implementation notes:

FreeRDP frame capture now produces raw BGRA frame bytes for worker runtime render notifications.
SessionRuntime splits render publication into two outputs:
- direct binary render sink receives raw frame bytes
- compatibility fallback sink builds JSON/base64 session_frame only for backend gateway/live-state fallback
Worker direct WSS sends raw captured frame bytes as RAP2 binary WebSocket frames when render_transport=binary_v1 is active.
Backend gateway fallback remains unchanged and still receives JSON/base64 session.frame compatibility events.
Direct render remains latest-frame-only and droppable; input/control scheduling is unchanged.

Smoke proof:

direct session id: b4720057-db61-4c72-bb4c-bccfd7e30008
fallback session id: 65d0667b-aaef-4042-ae30-4c34d151e5aa
direct client selected direct_worker_wss with render_transport=binary_v1
fallback client selected backend_gateway
direct client received binary_frame_received payloads of 3,686,400 bytes for 1280x720 BGRA
direct client rendered frame sequences including 1, 2, 4, 7, 9, 13, 14, 15, 16, 18, 19, and 20
fallback client rendered JSON/base64 session.frame through backend gateway
worker logs show raw_frame_bytes=3686400, binary_direct_bytes=3686400, base64_compat_bytes=4915200, encode_skipped_for_direct=true, and fallback_compat_frame_built=true
worker direct outbound logs show binary_render_bytes_per_second non-zero and json_render_bytes_per_second=0.000000

Payload and conversion proof:

direct raw frame payload remains 3,686,400 bytes plus the small RAP2 preamble/header
fallback compatibility payload remains about 4,915,200 base64 bytes for the same frame
direct render no longer decodes frame_data from compatibility base64 before binary send
base64 is still generated for fallback/debug because the backend gateway path intentionally remains JSON/base64

Known limitations:

DP-2.1 is an internal worker render plumbing cleanup only.
Full-frame BGRA payloads are still heavy.
At the DP-2.1 point, dirty regions, tiles, adaptive quality, compression/codecs, and color-mode reduction remained future work.
Backend gateway fallback remains JSON/base64 by design.

Stage DP-3A: Grayscale Full-Frame Binary Render

Status: implemented and smoke-proven on the test Docker environment as of 2026-04-25.

DP-3A adds the first conservative quality foundation for the direct binary render path. DP-3A itself did not implement tiles, compression, codecs, or adaptive profile switching. Dirty-region direct binary rendering is handled by the later RDP Adapter RDP-Perf-6 path.

Contract changes:

backend direct binary render candidates advertise render_transport=binary_v1
backend direct binary render candidates advertise supported_color_modes=["full_color","grayscale"]
backend direct binary render candidates advertise default_color_mode="full_color"
Windows client requests full_color by default
Windows smoke can request grayscale through -DirectDataPlaneColorMode grayscale
RAP2 binary frame headers carry color_mode, quality_profile, original_frame_format, output_frame_format, raw_frame_bytes, and binary_direct_bytes

Implementation notes:

full_color direct render sends the existing raw BGRA frame unchanged.
grayscale direct render converts BGRA bytes in the worker direct binary sink before WSS send.
The grayscale path preserves BGRA32 output format so the Windows presenter can reuse the existing render path.
Backend gateway fallback remains JSON/base64 and is not affected by direct grayscale mode.
Direct render remains latest-frame-only and droppable.
Input/control scheduling is unchanged and remains higher priority than render.

Smoke proof:

direct full-color session id: 74a0e5c6-02e0-487f-a1a1-c2850a13881c
direct grayscale session id: 3b616bd7-1179-4ec5-879f-7cd270f92a0a
fallback backend-gateway session id: e5724cac-7f09-4931-9ad9-156a3f33d0b1
direct full-color client selected direct_worker_wss with render_transport=binary_v1, requested_color_mode=full_color, and applied_color_mode=full_color
direct grayscale client selected direct_worker_wss with render_transport=binary_v1, requested_color_mode=grayscale, and applied_color_mode=grayscale
fallback smoke selected backend_gateway and continued to render JSON/base64 session.frame events
direct full-color frames rendered with color_mode=full_color and bytes=3686400
direct grayscale frames rendered with color_mode=grayscale and bytes=3686400
worker logs show grayscale_conversion_applied=false for full color and grayscale_conversion_applied=true for grayscale
worker logs show raw_frame_bytes_before=3686400, raw_frame_bytes_after=3686400, and binary_direct_bytes=3686400
worker grayscale conversion time was observed around 1-2 ms per sampled 1280x720 BGRA frame
worker direct outbound logs show binary render traffic and json_render_bytes_per_second=0.000000 on the direct binary path

Verification commands:

pwsh -ExecutionPolicy Bypass -File scripts/windows-smoke/desktop-smoke.ps1 -PreferDirectDataPlane:$true -AllowInsecureDirectDataPlaneTlsForSmoke:$true -DirectDataPlaneConnectTimeoutMs 2500 -DirectDataPlaneColorMode full_color -SkipOrgSwitchAndTokenRefresh
pwsh -ExecutionPolicy Bypass -File scripts/windows-smoke/desktop-smoke.ps1 -PreferDirectDataPlane:$true -AllowInsecureDirectDataPlaneTlsForSmoke:$true -DirectDataPlaneConnectTimeoutMs 2500 -DirectDataPlaneColorMode grayscale -SkipOrgSwitchAndTokenRefresh
pwsh -ExecutionPolicy Bypass -File scripts/windows-smoke/desktop-smoke.ps1 -PreferDirectDataPlane:$false -AllowInsecureDirectDataPlaneTlsForSmoke:$true -DirectDataPlaneConnectTimeoutMs 2500 -DirectDataPlaneColorMode grayscale -SkipOrgSwitchAndTokenRefresh

Known limitations after DP-3A:

grayscale currently reduces color fidelity but not wire byte size because the output format remains BGRA32.
256_colors, 64_colors, 16_colors, and palette modes are not implemented.
Tiles, compression/codecs, and adaptive profile switching remain future work.
Backend gateway fallback remains JSON/base64 by design.
Smoke rendering=false remains a compact-layout harness artifact in some runs; client logs prove Desktop frame received and SessionWindow rendered frame.

RDP-Perf-6: Direct Dirty-Region Binary Render Contract

Status: implemented and build/probe/live-smoke-proven on the test Docker environment as of 2026-04-26 using P3.3 Secret RDP Resource, direct worker WSS, and rap-rdp-worker:rdp-perf6-dirty-region.

RDP-Perf-6 keeps the existing RAP2 binary WebSocket transport and adds explicit direct render message types:

render.frame.full
render.frame.region

Compatibility:

Windows client direct transport still accepts legacy binary message_type=session.frame.
Inside the Windows application pipeline, direct binary frames are normalized back into the existing session.frame envelope so UI, lifecycle, input, clipboard, and file transfer behavior remain unchanged.
Backend gateway fallback remains JSON/base64 and is not removed.

Dirty-region frame metadata:

frame_width, frame_height, frame_stride, frame_format
desktop_width, desktop_height
region_x, region_y, region_width, region_height
region_stride, region_format=BGRA32
payload_length
input_correlation_id and worker_frame_captured_at when available

Diagnostics added for payload and latency analysis:

full_frame_sent
region_frame_sent
full_frame_bytes
region_bytes
region_savings_percent
diff_time_ms
render_update_reason
fallback_to_full_frame_reason

Implementation notes:

Worker direct WSS emits render.frame.full for baseline/recovery frames and render.frame.region for dirty-region patches.
Worker direct render logs include payload savings and diff/capture timing.
Windows direct transport accepts the explicit render message types.
Windows DesktopFramePresenter maintains a session framebuffer and patches BGRA32 region payloads into it before presenting the updated surface.
Full-frame fallback remains available for first frame, attach/reattach, resize, region-loss repair, and debug/fallback paths.

Observed runtime proof:

Direct transport selected direct_worker_wss with render_transport=binary_v1.
Baseline frame used render.frame.full, 1280x720, 3,686,400 bytes.
Dirty-region examples used render.frame.region: 64x64 = 16,384 bytes (99.56% savings), 1280x128 = 655,360 bytes (82.22% savings), and 640x64 = 163,840 bytes (95.56% savings).
Direct-only binary region frames logged fallback_compat_frame_built=false while backend gateway fallback compatibility remained available separately.
Input, detach, reattach, takeover, and takeover event handling remained smoke-proven in the same run.

Stage DP-3B: Adaptive Quality

Implement quality profiles and adaptive render behavior.

Goals:

lower latency under load
bounded queues
real profile behavior
color mode adaptation

12. Risks

Token Leakage

Risk:

direct worker token could be reused.

Mitigation:

short TTL
jti / nonce
worker-scoped audience
attachment/session binding
TLS

Worker Endpoint Exposure

Risk:

worker direct endpoint becomes an attack surface.

Mitigation:

token validation before bind
rate limits
TLS
no unauthenticated session enumeration
minimal endpoint surface

Policy Drift

Risk:

backend and worker disagree on allowed channels.

Mitigation:

token claims include allowed channels
worker receives policy snapshot in assignment
worker enforces policy again
policy changes trigger session update or reconnect where required

Fallback Masking Production Problems

Risk:

clients silently fall back and hide direct data-plane failure.

Mitigation:

log fallback reason
expose telemetry
smoke tests verify both direct and fallback paths

Render Still Too Heavy

Risk:

direct WSS improves routing but full-frame render remains expensive.

Mitigation:

DP-2 binary frames
DP-3 adaptive quality
dirty regions / tiles
latest-frame-only semantics

File Upload Starving Input

Risk:

reliable file chunks can fill send queues.

Mitigation:

channel priority
bounded file queues
chunk pacing
input preemption

13. Future Verification Plan

Future DP-1 implementation must prove:

backend gateway fallback still works
direct worker WSS connects
token validation works
invalid/expired/wrong-worker tokens are rejected
direct WSS binds to existing session runtime
direct WSS does not recreate remote RDP session
input works over direct WSS
rendering works over direct WSS
clipboard still works
file upload still works
fallback activates if direct worker path is unavailable
input latency improves compared with fallback
render backlog does not grow
stale render frames are dropped
close/dispose is immediate
org/session/attachment/channel scope is enforced

14. Next Implementation Prompt

Data-plane and RDP work are paused by product decision.

DP-3B, Stage 5.2 remaining RDP desktop proof, and further RDP performance work must not start without a new explicit RDP/data-plane stage prompt.

The next active project work is Stage C10 in the lower Secure Access Fabric foundation:

Proceed with Stage C10 only.

Goal:
Consolidate Fabric Core architecture and prepare scoped cluster configuration
distribution design.

Scope:
- define signed scoped cluster snapshot model
- define node-local state boundaries
- define peer directory/cache boundaries
- define Fabric Storage / Config Storage role
- define source-of-truth vs distribution/cache boundaries
- define multi-cluster isolation boundaries
- define future implementation stages C11-C18

Do NOT:
- implement mesh runtime
- implement VPN
- implement RDP work
- implement service workloads
- change backend/runtime code

15. Non-Goals

DP-1 does not implement:

full mesh
VPN
QUIC
UDP transport
WebRTC
relay nodes
multi-cluster routing
adaptive quality beyond DP-3A grayscale full-frame foundation
binary render frames for fallback backend gateway
adaptive profile switching beyond DP-3A and dirty regions
removal of current backend WebSocket gateway
RDP MVP rewrite

44 KiB Raw Blame History

Data Plane v1 for RDP

1. Current Problem

2. Target DP-1 Path

3. Responsibilities

Backend as Control Plane

Worker as Direct Realtime Endpoint

Windows Client

4. Backend Contract Proposal

Proposed DTO Shape

5. Data Plane Token Model

6. Direct WSS Channel Model

control

input

render

clipboard

file_upload

telemetry

7. Message Framing

JSON Envelope

Binary Frame Header

8. Quality Profile Foundation

9. Security Model

10. Fallback Backend Gateway Path

11. Migration Stages

Stage DP-1A: Spec Only

Stage DP-1B: Backend Offers Data Plane Candidates

Stage DP-1C: Worker Direct WSS Endpoint

Stage DP-1D: Windows Client Prefers Direct WSS

Stage DP-1D.1: Worker Direct JSON Realtime Bridge

Stage DP-1E: Latency Comparison

Stage DP-2: Binary Render Frames

Stage DP-2.1: Worker Raw-Frame Split

Stage DP-3A: Grayscale Full-Frame Binary Render

RDP-Perf-6: Direct Dirty-Region Binary Render Contract

Stage DP-3B: Adaptive Quality

12. Risks

Token Leakage

Worker Endpoint Exposure

Policy Drift

Fallback Masking Production Problems

Render Still Too Heavy

File Upload Starving Input

13. Future Verification Plan

14. Next Implementation Prompt

15. Non-Goals

44 KiB

Raw Blame History

`control`

`input`

`render`

`clipboard`

`file_upload`

`telemetry`