Skip to content

Training Tab

Overview

The Training tab (TrainingPage.qml, route Training) orchestrates plugin-based semantic-segmentation training runs (U-Net, DeepLabV3+, SegFormer-B2, etc.) on either local Docker compute or authorized Remote SSH GPU nodes, displaying live validation Dice scores, epoch progress, pause/resume controls, and detailed training diagnostics.

Training runs table displaying Model Name, Dataset, Architecture & Encoder, Epoch progress, Validation Dice, Status badges, and pause/resume actions.

Availability & Prerequisites

  • Ready Dataset: At least one dataset in Ready state from the Datasets tab.
  • Execution Target:
    • Profiles production / stage (Modes 3–4): Enforce Authorized remote worker (ssh_node).
    • Local / Mode 2 profiles: Offer both Local Docker (local) and Remote SSH Node (ssh_node).

Step-by-Step User Workflows

1. Launch a New Training Job

  1. Open the New training dialog (TrainingJobDialog, 680px wide) from the page action button.
  2. Wait for Checking training-ready datasets… to reach ready (click refresh if a dataset finished recently).
  3. Select a Dataset (<dataset_name> · <dataset_id> v<version>), choose a Plugin / Architecture (U-Net, DeepLabV3+, SegFormer-B2), enter a Model Name, and choose an Encoder (e.g., resnet18).
  4. Configure hyperparameters including epochs, Augmentation profile (None, Basic, Heavy, Histology focused), and optional max_runtime_seconds.
  5. Choose the Execution target (Local Docker or Remote SSH Node -> select node showing [status (FIFO queue available)] or [status (Full - Queued)]).
  6. Submit to queue the job. A "Training queued" toast displays the created job_id.
New training dialog modal with dataset selection, augmentation profiles, remote worker SSH target, and plugin architecture options.

2. Monitor, Pause & Resume Portable Checkpoints

  • Live Sync: Active training jobs update status, current_epoch / total_epochs, and val_dice automatically via TrainingSyncService (2-second active cadence / 5-second idle cadence).
  • Inspect Details (TrainingDetailsModal): Click a job row to examine full configuration, epoch metric history, curves, and artifact download options.
  • Pause (training_pause): Available on running jobs whose worker supports portable checkpoints (training_resume_enabled === true). Clicking pause sets stage to pause_requested ("Waiting for the current epoch to finish").
  • Resume (training_resume): Available on paused, failed, or timed_out jobs via trainingResumeDialog to continue training from the saved portable checkpoint.
stateDiagram-v2
    [*] --> queued: Pending
    queued --> running: Worker boots plugin
    running --> pause_requested: User clicks Pause
    pause_requested --> paused: Current epoch finishes & checkpoint saved
    paused --> running: User resumes via trainingResumeDialog
    failed --> running: User resumes portable checkpoint
    running --> completed: All epochs finish (Model registered)
    running --> failed: Error / OOM
    running --> timed_out: Exceeded max_runtime_seconds
    running --> cancelled: User cancels / force stops

UI Controls & Status Reference

Supported Architectures & Augmentation Profiles

Setting Available Options / Examples Notes
Model Architectures U-Net, DeepLabV3+, SegFormer-B2 (from registered plugins) Resolved dynamically from trainingViewModel.availablePlugins
Augmentation Profiles None (none), Basic (basic), Heavy (heavy), Histology focused (histology_focused) Applied strictly to the training split; validation split remains deterministic
Execution Targets Local Docker (local), Remote SSH Node (ssh_node), Authorized remote worker (production/stage) Production/Stage profiles restrict execution to authorized remote workers

Job State & Badge Semantics

Internal state / stage UI Badge Label Badge Color Pause / Resume Eligibility
stage === "pause_requested" Pause requested Amber (#D97706) Pause disabled ("Waiting for the current epoch to finish")
queued Pending Amber (#D97706) —
running Running Blue (#2563EB) Pause enabled when training_resume_enabled === true
paused Paused Gray (#6B7280) Resume enabled
completed / succeeded Completed Green (#059669) Download verified .zip model package
failed / timed_out Failed / Timed out Red (#DC2626) Resume available if portable checkpoint exists
cancelled / cancel_requested Cancelled Muted Gray (#9CA3AF) Terminal

Metrics Resolution Precedence (val_dice)

The Training table renders the best available Validation Dice value (4 decimal places) in this order: metrics.val_dice -> metrics.dice -> metrics.best_val_dice -> latest metrics_history entry -> job.validation_dice (or — if unavailable).

Dialogs & Modals

Dialog Component Behavior
New training TrainingJobDialog (trainingJobDialog) Prevents outside-click/Escape dismissal while submitting (starting_training or preparing_dataset)
Training Details TrainingDetailsModal (trainingDetailsModal) Opens on row selection; shows hyperparameters, loss/Dice history, logs, and model download
Resume Training trainingResumeDialog Restarts paused, failed, or timed_out jobs from saved checkpoint
Plugin Release Error pluginReleaseErrorDialog Triggered via onOperationFailedDetails(message, details) if plugin resolution/release mismatch occurs

Technical Reference

  • Worker Execution Contract: Training executes inside an isolated worker process receiving a versioned job envelope and scoped temporary credentials; the worker reports bootstrap, heartbeat, per-epoch metrics, and terminal events back to the backend.
  • Monitoring Direction & Checkpoints: Best checkpoints and final portable model archives remain tied to exact plugin ID/version and runtime bindings for reproducibility in Models.