Managing and Promoting Trained Models

After you've exported an approved dataset, the next steps are training a model on it and then promoting that model through a governed lifecycle to production. StretchGPT's model registry enforces that no model reaches staging or production without a recorded evaluation and a human sign-off. This article walks the whole path. Dispatching training, running evals, and promoting all require step-up admin authorization.

Dispatching a training job

1. In the training console, choose Dispatch training job.
2. Configure it:
- Job type — for example, a LoRA train.
- Family — which assistant family this model serves.
- Dataset — the approved dataset to train on.
- Base model — the model to fine-tune from.
- Params, priority, and max attempts as needed.
3. Optionally set an idempotency key so a retried dispatch doesn't create duplicate jobs.
4. Authorize. The job is queued for a GPU worker node to lease and run (see Worker Nodes and the Safety Envelope).

You can list training jobs and filter by status to watch progress. When a node finishes a lora_train job successfully, the result is recorded and a model artifact enters the registry in the trained state.

The model lifecycle

Models move through a strict set of states. A model becoming trained does not make it production-eligible — governance gates every step:

candidate → trained → evaluated → internal_test → staging → production_eligible
                    ↘ internal_test ↗
(any state) → rejected

• candidate → trained: training finished.
• trained → evaluated or internal_test: you begin assessing it.
• evaluated / internal_test → staging: promotion into staging — requires a recorded eval run and a sign-off.
• staging → production_eligible: the final gate — again requires an eval and sign-off.
• Any state → rejected: retire a model that doesn't make the bar.

Invalid jumps (for example, straight from trained to production_eligible) are refused.

Evaluating a model

1. Open the model in the registry and choose Run eval.
2. Authorize. StretchGPT dispatches an evaluation job for the model.
3. When it completes, the eval is recorded against the model. You can open the model to see its evaluation history.

At least one recorded eval run is mandatory before staging or production — the registry will block promotion without it.

Promoting a model

1. Open the model and choose Promote.
2. Select the target state (for example, staging).
3. Provide your sign-off and any notes.
4. Authorize. If the transition is allowed and the eval-and-sign-off requirements are met, the model moves to the new state; the promotion time and who signed off are recorded.

If you try to stage or promote without an eval or without a sign-off, StretchGPT refuses and tells you which requirement is missing.

A realistic example

Your "Support tone v2" dataset is ready. You dispatch a LoRA train against a base model for the support family. A GPU node picks it up, trains, and reports success — the adapter lands as trained. You run an eval; the report is recorded. Satisfied with the numbers, you promote it to staging with a sign-off note ("tone eval passed, 3% win rate over prod"). After a soak in staging and a second eval, you promote it to production_eligible, again signing off. Only now can it serve real traffic — and there's a full record of every gate it passed.

Tips

• Use idempotency keys on dispatch so a network retry doesn't double-queue a job.
• Evaluate before every promotion gate, not just once — staging and production each require it.
• Write meaningful sign-off notes. They're the governance record for why a model shipped.
• Reject decisively. Retiring a weak adapter keeps the registry honest.

Troubleshooting

Promotion refused: eval required. Staging and production need at least one recorded eval run. Run an eval, then promote.

Promotion refused: sign-off required. You must provide a sign-off to stage or promote. Add it and retry.

Invalid transition. You tried to skip a state (for example, trained → production). Move through the allowed states in order.

My training job never runs. No GPU worker node is leasing jobs of that type. Confirm a node is registered, polling, and scoped for the job type.