The label-budget ledger: count training, selection and calibration together

01 / Why is the training count not enough?

Two methods using 32 training examples do not necessarily share the same supervision condition. If one method repeatedly uses a larger validation set to choose epochs, thresholds and fusion weights, it relies on more labeled information. A low-resource comparison needs one ledger for training, selection and calibration.

02 / What belongs in the ledger?

Each budget unit should record the dataset, class, sample ID, purpose, first accounting stage, whether it was reused during development and whether it belongs to the independent test. Reusing an already labeled sample during development is not charged twice, while labels added for new classes accumulate by stage.

ledger = [
    {"sample_id": "meld-001", "stage": "train", "labels": 1},
    {"sample_id": "meld-009", "stage": "selection", "labels": 1},
    {"sample_id": "meld-014", "stage": "calibration", "labels": 1},
]

# Count distinct labeled sample IDs per protocol,
# then keep independent-test labels separate.
used = {row["sample_id"] for row in ledger}
print(len(used))

03 / How do we keep methods comparable?

Share data indices, random seeds, tuning trials and validation resamples. After fixing the budget grid, run paired repeats. Record Macro-F1, UAR, class-wise recall, prediction proportions and training cost together instead of letting one best score represent all evidence.

04 / Relation to the earlier study

The Neurocomputing revision limits training labels while model selection still uses a fully labeled validation set. The proposed study moves this boundary forward by counting selection and calibration in the total budget. It motivates the question; it is not a new experiment result.

05 / Checklist before the final test

• Freeze the budget protocol, sample indices and label mapping.
• Keep independent-test labels out of selection, calibration and threshold choice.
• Report performance, dispersion and resource cost as the budget changes.