Model evaluation: stopping, alignment and ensembles

01 / Which model are we selecting?

Stopping training at epoch ten does not imply using the parameters from epoch ten. Early stopping decides when to stop observing; the best epoch identifies the candidate to retain. Before discussing scores, specify the validation metric, its direction, the minimum improvement and patience.

Training data updates parameters; validation data selects the epoch, threshold and ensemble weights. Evaluate an independent test set after fixing those choices. Repeatedly inspecting test results and changing the design makes that set part of selection, weakening its role as independent evidence of generalization.

02 / How the source defines improvement

In max mode, EarlyStopping requires a score strictly greater than best_score + min_delta; equality does not count. For validation scores 0.60, 0.64, 0.63 and 0.64, with patience=2 and min_delta=0, the fourth observation triggers stopping while epoch two remains best.

The implementation rejects nonfinite scores and nonincreasing epochs. Once stopped, it stays stopped until reset. Patience counts validation observations. The class records decisions but neither saves nor restores model weights, so a training adapter still needs checkpoint persistence.

03 / Equal row counts do not establish identity

ensemble_predictions retrieves values by sample_id and checks that every table contains the same ID set. The CSV reader also rejects duplicate IDs. Two files can have equal row counts while row-wise averaging mixes different samples; array shape checks cannot detect this.

Inputs must be finite probabilities in [0,1] for the same positive class, not raw logits. Weights must be nonnegative with at least one positive entry. ID alignment resolves identity, but it neither calibrates probabilities nor repairs inconsistent meanings of label 1.

04 / A small example to inspect step by step

This simplified teaching snippet runs after installing the public KAPT package. The second prediction table deliberately reverses its order: the result should still be s1=0.75 and s2=0.25. The assertions also distinguish the stopping epoch from the best epoch.

# Simplified teaching example; install the KAPT package first.
from kapt_lab.early_stopping import EarlyStopping
from kapt_lab.ensemble import ensemble_predictions

stopper = EarlyStopping(patience=2, mode="max")
stops = [stopper(score, epoch) for epoch, score in
         enumerate([0.60, 0.64, 0.63, 0.64], start=1)]
assert stops == [False, False, False, True]
assert stopper.best_epoch == 2

a = {"s1": 0.9, "s2": 0.4}
b = {"s2": 0.1, "s1": 0.6}
combined = ensemble_predictions([a, b])
assert combined == {"s1": 0.75, "s2": 0.25}
print(stopper.best_epoch, combined)

This checks a program contract for fixed inputs. It trains no model and does not establish that ensembling improves a real task.

05 / Make evaluation inspectable

A mean of metrics describes individual models; ensemble metrics require averaging predictions first and then evaluating. This site's build_metrics.py uses score >= threshold and records the input hash and library version. Changing the threshold changes confusion counts, but not the ROC AUC of a fixed score vector.

• Reordering prediction rows should preserve results; missing, extra or duplicate IDs should fail explicitly.
• Check tied scores, the min_delta boundary, nonfinite inputs and the state after stopping.
• Retain per-sample probabilities and confusion counts. With single-class labels, the package returns null for AUC and AP, while the demo generator rejects that scenario.

06 / Limits and the next question

Synthetic fixtures can expose alignment bugs, but cannot establish effectiveness under distribution shift, class imbalance or calibration error. A next study would fix splits in advance, compare individual models with equal-weight ensembles, and document failure cases and uncertainty across runs. KAPT is a learning package curated from a KANO-based workspace; the upstream method and paper belong to their original authors.