A concrete first study

01 / One question

Under the same total development-label and compute budgets, does frozen visual–textual fusion improve emotion classification over the stronger unimodal baseline?

This first study addresses the budget question in the existing RP. Selective decisions and continual updates follow only after the baseline protocol is usable. Dataset and model choices are not yet finalized.

02 / Design and controls

1. Fix the data protocol

   Select one permitted emotion dataset with visual segments and transcripts. Freeze label mapping, official or speaker/session-disjoint splits, near-duplicate checks, preprocessing and pretrained encoder provenance before model comparison.

2. Count all development labels

   Proposed initial grid: 16, 32 and 64 labels per class, divided 50%/25%/25% among training, model selection and calibration, with disjoint IDs. Confirm feasibility before freezing the protocol; log unused allocations. Separately disclose the fixed test-label cost.

3. Start with simple comparisons

   Use text-only, vision-only and concatenated frozen-feature linear classifiers. Add a small MLP as the first richer model. Select the stronger unimodal comparator on development data; share splits, sampled IDs, five paired seed schedules and search-trial caps across methods.

4. Compare effectiveness and cost

   Primary endpoint: paired Macro-F1 difference. Also report UAR, per-class recall, predicted class proportions, variability, training/selection cost and batch-1 latency on specified hardware. Keep the test set locked until choices are fixed.

03 / How to interpret the result

Report each budget separately. Use the same predeclared uncertainty procedure for the primary comparison; cluster resampling by speaker or session where observations are dependent, and describe how seed variation is summarized.

An interval spanning zero is inconclusive evidence of improvement, not proof that the methods are identical. If gains vanish after budget or duplicate controls, report that outcome and inspect the source of the earlier gain. Do not change the primary endpoint after viewing test results.

04 / First deliverables

• A frozen protocol, label ledger, data manifest and preprocessing configuration.
• A single command for each baseline, environment lock and exact commit reference.
• Per-example scores and permitted sample IDs, with selection/calibration decisions logged.
• An error-analysis page showing where fusion helps, fails or has insufficient evidence; share derived artifacts within the dataset license.

All items on this page are a discussion protocol, not completed emotion-recognition results or a fixed doctoral commitment. The first milestone is an auditable baseline comparison.