Evidence-aware multimodal decisions: from quality signals to review

01 / When should automatic output be withheld?

A high confidence score only says that the model assigns high probability to a class. It does not establish that the input evidence is complete or that modalities agree. A better interaction entry point is to separate evidence states before deciding whether automatic output is allowed.

02 / What does each signal tell us?

Quality proxies describe input conditions such as audio signal quality, visual detection quality or transcript confidence. Missing masks record availability. Cross-modal disagreement describes prediction conflict, while calibrated confidence describes the relationship between probability and correctness. These signals are not interchangeable.

• Low quality does not guarantee an error; high quality does not guarantee agreement.
• When every modality is missing, the system should enter an explicit insufficient-evidence state instead of returning a normal-looking class.

03 / How does a minimal gate work?

Choose thresholds on development data. Allow automatic output only when the highest calibrated probability reaches its threshold, disagreement stays below its threshold and at least one modality is available; otherwise request review. The snippet below demonstrates state logic only.

function decide({quality, confidence, disagreement, available}) {
  if (!available.some(Boolean)) return {state: "insufficient-evidence"};
  if (quality < 0.45) return {state: "review", reason: "quality"};
  if (confidence < 0.70) return {state: "review", reason: "confidence"};
  if (disagreement > 0.45) return {state: "review", reason: "disagreement"};
  return {state: "auto-output"};
}

// This is a contract example, not a fitted classifier.

04 / How would we check whether the gate helps?

Overall accuracy is insufficient. Under missing modalities, noise, occlusion and transcript perturbations, compare risk at the same coverage or coverage at the same risk, and report ECE. Remove quality, disagreement and calibration signals separately to see which mechanism changes the decision boundary.

The online gate uses synthetic values to make state transitions readable. It does not establish a real-task risk–coverage result.

05 / Boundary for the next study

A real study must freeze data splits, label budgets, threshold rules and an independent-test protocol before evaluation, then report datasets separately. Unknown-class detection and open-vocabulary recognition also need separate protocols rather than being folded into the known-class degraded-input metric.