Worker Nodes and the Training Bridge

The heavy compute in StretchGPT — training and evaluation — runs on GPU worker nodes, not on the control plane. Nodes connect with scoped, single-purpose credentials, pull work over outbound HTTPS, and are protected against replay. This article explains how nodes register and operate, and the guarantees that keep the lane secure.

The outbound-polling model

A StretchGPT worker node never opens an inbound listener. Instead it reaches out to the control plane over HTTPS to register, send heartbeats, poll for jobs, and submit results. This means a GPU box — even one on a home or lab network — can do real work without exposing any port to the internet. Registration returns a recommended poll interval (about every 5 minutes) so nodes stay in touch without hammering the server.

Scoped node credentials

Every node action is gated by a node credential carrying a specific scope. A credential is minted for exactly what a node needs to do:

• node:register — announce the node and its capabilities.
• node:heartbeat — report liveness and telemetry.
• node:poll — lease the next job.
• node:result:submit — return a completed job's result.
• node:audit:submit — submit audit events.

A credential scoped only to heartbeat can't lease jobs or submit results — least privilege by construction. Credentials are issued and verified by the control plane; the node presents its token on every call.

Registering a node

1. Provision the node with a credential scoped for node:register.
2. The node calls register, sending its node ID, display name, role (typically a local GPU compute node), and its capabilities and governor limits.
3. The control plane records the node as registered, stamps it with a policy version, and returns the next poll interval.
4. The node then heartbeats and polls on schedule under its heartbeat/poll credentials.

The training bridge and replay protection

For the real training lane, node requests carry two extra guards on top of the scoped credential:

• Timestamp freshness — each request includes a signed timestamp that must fall inside a 5-minute window. Stale requests are refused.
• Single-use nonce — each request includes a nonce that can be used only once. A replayed nonce is rejected as a replay.

Together these mean a captured request can't be replayed later to trigger duplicate work. A node leases the next job, heartbeats progress while it works (extending its lease), and submits the result — all under its scoped credential with fresh timestamp-and-nonce on every call.

Safe vs. blocked job types

The job lane distinguishes safe job types (dry-run and read-only work such as status checks, telemetry sync, dataset-prepare dry runs, and embedding dry runs) from blocked job types (confirmed teacher writes, raw training, auto-promotion, dataset mutation, model-file writes, and public inbound listeners). The blocked set can only run when the runtime is explicitly activated — see The Safety Envelope and Execution Gate.

A realistic example

You have a workstation with a capable GPU. You provision it with node credentials and it registers as "Studio-GPU," reporting its VRAM and governor limits. It begins polling every few minutes. When you dispatch a lora_train job, the node leases it, heartbeats progress as it trains, and on completion submits the result — which lands your new adapter in the registry as trained. Throughout, every call carried a fresh timestamp and a one-time nonce, so nothing the node sent could be replayed.

Tips

• Mint the narrowest credential. Give a node only the scopes it actually uses.
• Keep the node's clock in sync. The 5-minute timestamp window means clock drift will get requests rejected.
• Let nodes poll, don't push. The outbound model is intentional — no inbound port needed.
• Watch heartbeats. A node that stops heartbeating has lost its lease; the job becomes available again.

Troubleshooting

Node requests fail with a timestamp error. The node's clock is likely off or the request is stale. Sync the clock and ensure each request sends a fresh signed timestamp within the 5-minute window.

Node requests fail as a replay. The nonce was reused. Each request needs a unique, single-use nonce.

Node can register but can't lease jobs. Its credential is scoped for register/heartbeat but not node:poll. Issue a poll-scoped credential.

Jobs stay queued forever. No node is polling for that job type, or the runtime isn't activated for training. Confirm a node is online and that training is enabled in the safety envelope.