Panel-data validation: keep the whole forecast horizon on one side of a split

01 / More rows do not mean more independent evidence

Thirty provinces over twenty years would yield 600 province-year rows only if the panel were complete. Revisions, missingness, target construction and lag windows can reduce the usable set. Nearby provinces and adjacent years are correlated, so 600 rows do not imply 600 independent experiments.

Before counting samples, specify whether one example is a province-year, a time window, a network snapshot or a forecast horizon. Save an identifier such as province + origin + horizon to align predictions and prevent accidental duplicates.

02 / The test split must answer the intended question

A future-time test on already observed provinces measures temporal generalization. Holding out provinces measures transfer to new regions. Holding out both is harder and addresses a third question. Report these protocols separately rather than treating one as evidence for all three.

For graph models, state whether held-out nodes, their features and their edges are visible during training. Transductive access to an unlabeled node is different from inductive prediction on a previously unseen node; labels must remain withheld in either case.

03 / Check label intervals, not just start dates

In an illustrative split, training labels end in 2016, validation labels cover 2017–2020 and test labels cover 2021–2024. A two-year forecast issued in 2015 targets 2016 and 2017, crossing the training/validation boundary. Purge it under this strict label-disjoint protocol.

A validation window may use earlier observed history: sharing historical inputs is not automatically leakage. Check whether a fitted transformation, target, revised statistic or graph edge introduces information unavailable at the origin. The illustrative dates below are not claimed to be the paper’s split.

from climate_protocol import split_window

assert split_window(2014, 2) == 'train'       # 2015, 2016
assert split_window(2015, 2) == 'purged'      # 2016, 2017
assert split_window(2016, 2) == 'validation'  # 2017, 2018
assert split_window(2019, 2) == 'purged'      # 2020, 2021
assert split_window(2020, 2) == 'test'        # 2021, 2022

04 / Preprocessing is part of the model

Fit imputers, scalers, feature selectors and learned graph construction on training data only. Apply the frozen transformations to validation and test data. If a procedure uses a future value to interpolate a historical gap, it can leak even when its code is outside the model.

• Lock target definitions, baseline rules and metrics before final testing. Use validation for model selection; repeated test-guided tuning turns the test into development data.
• Report error by horizon and region, not just a pooled average. For rare extreme events, inspect event counts and error distribution.
• Use dependence-aware uncertainty estimates where feasible; resampling individual province-year rows may understate uncertainty when errors cluster.

05 / What the executed check establishes

The script enumerates two-year windows and checks that retained label-year sets are disjoint across splits. Ten focused tests also cover invalid horizons, information availability, graph inputs and quantile arithmetic. They do not check spatial holdouts, measurement quality or model accuracy.

python -m unittest discover -s notebook/examples -p test_climate_protocol.py -v

06 / Write the split before building the model

A useful first artifact is a table of issue time, available features, label interval, region set and split. If a colleague cannot reconstruct which information each prediction used, improving the model architecture will not repair the evaluation protocol.