The Training tab (TrainingPage.qml, route Training) orchestrates plugin-based semantic-segmentation training runs (U-Net, DeepLabV3+, SegFormer-B2, etc.) on either local Docker compute or authorized Remote SSH GPU nodes, displaying live validation Dice scores, epoch progress, pause/resume controls, and detailed training diagnostics.
Training runs table displaying Model Name, Dataset, Architecture & Encoder, Epoch progress, Validation Dice, Status badges, and pause/resume actions.
Availability & Prerequisites
Ready Dataset: At least one dataset in Ready state from the Datasets tab.
Local / Mode 2 profiles: Offer both Local Docker (local) and Remote SSH Node (ssh_node).
Step-by-Step User Workflows
1. Launch a New Training Job
Open the New training dialog (TrainingJobDialog, 680px wide) from the page action button.
Wait for Checking training-ready datasets… to reach ready (click refresh if a dataset finished recently).
Select a Dataset (<dataset_name> · <dataset_id> v<version>), choose a Plugin / Architecture (U-Net, DeepLabV3+, SegFormer-B2), enter a Model Name, and choose an Encoder (e.g., resnet18).
Configure hyperparameters including epochs, Augmentation profile (None, Basic, Heavy, Histology focused), and optional max_runtime_seconds.
Choose the Execution target (Local Docker or Remote SSH Node -> select node showing [status (FIFO queue available)] or [status (Full - Queued)]).
Submit to queue the job. A "Training queued" toast displays the created job_id.
New training dialog modal with dataset selection, augmentation profiles, remote worker SSH target, and plugin architecture options.
2. Monitor, Pause & Resume Portable Checkpoints
Live Sync: Active training jobs update status, current_epoch / total_epochs, and val_dice automatically via TrainingSyncService (2-second active cadence / 5-second idle cadence).
Inspect Details (TrainingDetailsModal): Click a job row to examine full configuration, epoch metric history, curves, and artifact download options.
Pause (training_pause): Available on running jobs whose worker supports portable checkpoints (training_resume_enabled === true). Clicking pause sets stage to pause_requested ("Waiting for the current epoch to finish").
Resume (training_resume): Available on paused, failed, or timed_out jobs via trainingResumeDialog to continue training from the saved portable checkpoint.
stateDiagram-v2
[*] --> queued: Pending
queued --> running: Worker boots plugin
running --> pause_requested: User clicks Pause
pause_requested --> paused: Current epoch finishes & checkpoint saved
paused --> running: User resumes via trainingResumeDialog
failed --> running: User resumes portable checkpoint
running --> completed: All epochs finish (Model registered)
running --> failed: Error / OOM
running --> timed_out: Exceeded max_runtime_seconds
running --> cancelled: User cancels / force stops
Production/Stage profiles restrict execution to authorized remote workers
Job State & Badge Semantics
Internal state / stage
UI Badge Label
Badge Color
Pause / Resume Eligibility
stage === "pause_requested"
Pause requested
Amber (#D97706)
Pause disabled ("Waiting for the current epoch to finish")
queued
Pending
Amber (#D97706)
—
running
Running
Blue (#2563EB)
Pause enabled when training_resume_enabled === true
paused
Paused
Gray (#6B7280)
Resume enabled
completed / succeeded
Completed
Green (#059669)
Download verified .zip model package
failed / timed_out
Failed / Timed out
Red (#DC2626)
Resume available if portable checkpoint exists
cancelled / cancel_requested
Cancelled
Muted Gray (#9CA3AF)
Terminal
Metrics Resolution Precedence (val_dice)
The Training table renders the best available Validation Dice value (4 decimal places) in this order:
metrics.val_dice -> metrics.dice -> metrics.best_val_dice -> latest metrics_history entry -> job.validation_dice (or — if unavailable).
Dialogs & Modals
Dialog
Component
Behavior
New training
TrainingJobDialog (trainingJobDialog)
Prevents outside-click/Escape dismissal while submitting (starting_training or preparing_dataset)
Training Details
TrainingDetailsModal (trainingDetailsModal)
Opens on row selection; shows hyperparameters, loss/Dice history, logs, and model download
Resume Training
trainingResumeDialog
Restarts paused, failed, or timed_out jobs from saved checkpoint
Plugin Release Error
pluginReleaseErrorDialog
Triggered via onOperationFailedDetails(message, details) if plugin resolution/release mismatch occurs
Technical Reference
Worker Execution Contract: Training executes inside an isolated worker process receiving a versioned job envelope and scoped temporary credentials; the worker reports bootstrap, heartbeat, per-epoch metrics, and terminal events back to the backend.
Monitoring Direction & Checkpoints: Best checkpoints and final portable model archives remain tied to exact plugin ID/version and runtime bindings for reproducibility in Models.