From cleaning to annotation: account for every sample

01 · Cleaned numbers need traceable inputs

A result that only reports ten retained rows cannot explain what happened to the other inputs. The data preparation demo sends each source row to either the retained set or quarantine, recording sourceRow and reason. A source row number identifies a row; a business ID identifies an event. They are not interchangeable, and repeated IDs should not automatically be discarded as useless rows.

02 · Exact copies and conflicting content are different evidence

pipeline.mjs validates fields first, then groups by ID and compares normalized timestamps, groups and integer amounts. Amounts 1 and 1.00 can collapse into one record; amounts 2 and 3 under the same ID represent a conflict. The default quarantines every conflicting version. The alternative keep-first policy sets conflictRetained to expose its dependence on source order.

Keeping the latest version requires a trustworthy version field. Row order, collection time and business update time may differ. Column-based deduplication in Spark does not choose the correct business version on our behalf; the rule must come before the tool.

03 · Account separately for zero, missing and invalid values

Zero is an observation, an empty amount is unknown, and the text NaN is invalid input. The code isolates invalid values before applying a missing-value policy, so imputation cannot hide malformed input. Explicit zero imputation retains an imputed flag. Rejection reasons count source rows, so one conflicting ID may contribute two rejected records. Each row currently records its first failure reason, not a complete field-level diagnosis.

Checkable invariants: input rows = retained rows + quarantined rows; group totals in integer cents = the retained amount total. Intermediate stages are not yet fully deduplicated and must not be presented as final results.

04 · Extend conflict preservation to annotation

An annotation ledger should link sample IDs, label versions, annotators and review status. The synthetic exercise below only routes conflicting labels: agreement enters a candidate set and disagreement enters review. Keeping the first label does not establish ground truth. This snippet does not implement label validation, permissions or adjudication.

// Synthetic teaching example: agreement is not proof of truth.
function auditAnnotations(rows) {
  const groups = new Map();
  for (const row of rows) {
    if (!groups.has(row.id)) groups.set(row.id, []);
    groups.get(row.id).push(row);
  }
  const candidates = [], review = [];
  for (const [id, evidence] of groups) {
    const labels = new Set(evidence.map(row => row.label));
    if (labels.size === 1) candidates.push({ id, label: evidence[0].label });
    else review.push({ id, reason: "label_conflict", evidence });
  }
  return { candidates, review };
}
const result = auditAnnotations([
  { id: "s1", label: 1 }, { id: "s1", label: 0 },
  { id: "s2", label: 1 }, { id: "s2", label: 1 }
]);
console.log(result.candidates); // [{ id: 's2', label: 1 }]
console.log(result.review[0].reason); // label_conflict

05 · Passing cleaning does not make a sample model-ready

A collected record that passes format checks only satisfies the current contract. Model inputs still need fixed label semantics, dataset splits and source grouping; statistical imputation and normalization must be fitted on training data only. The evaluation script requires unique IDs, binary labels, finite scores and both classes, but it does not automatically align independent external files. Prediction joins must still be checked by ID rather than assumed from row positions.

06 · Test rules with counterexamples and question provenance

Existing tests cover invalid dates, zero versus missing values, duplicate conflicts, amount reconciliation and idempotence when cleaning retained data again. The annotation exercise can be checked with reversed order and repeated agreeing labels. Cross-split duplicate detection and adjudication logs should come before claims about annotation agreement. Seventeen synthetic rows cannot establish real annotation quality, collection compliance or distributed performance.