A1: Hedonic-Nutritional Correlation Analysis

Done Data Ready Low Effort Core

Research Question: How do hedonic ratings and social context interact with food choices and self-perceived healthiness?

Method: Mixed-effects regression on feeling × healthy × with × place × food_kind. Cluster participants by hedonic profiles. Visualize interaction effects.

Data Needed: MakanMemo cleaned CSV (6,166 rows with feeling, healthy, with, place, activity, food tags)

CHI Contribution: First large-scale quantitative evidence that hedonic/social context predicts dietary health perception in real-world meal logging. Challenges calorie-centric food tracking.

Novelty vs. Prior Work: Existing food PI work (Eat4Thought, FoodScrap) is qualitative or small-N. This is 6K+ meals from 249 people over 2 years.

CHI Session fit: Self-Tracking and Personal Health Informatics

Results

• Analysis set: 5,636 meals from 239 participants after dropping rows with missing context variables (530 dropped, 8.6%).
• Ceiling effect confirmed: 85.8% of feeling ratings are 4–5. Healthy ratings slightly less skewed (58.9% are 4–5).
• M1 (feeling ~ healthy + context): Self-perceived healthiness is the strongest predictor of hedonic feeling (β=0.151, p=7.2×10⁻⁸¹). Restaurant meals boost feeling (β=+0.126, p<0.001). Eating alone significantly lowers feeling (β=−0.071, p<0.001). Snacks are associated with higher feeling (β=+0.129, p<0.001). SMART participants report lower feeling than HAPPY (β=−0.154, p=0.021).
• M2 (healthy ~ feeling + context + food tags): Feeling strongly predicts perceived healthiness (β=0.316, p=4.0×10⁻⁶⁶). Food tags validate: sweets (β=−0.834) and deep-fried (β=−0.621) lower healthiness; fruits (β=+0.502) and vegetables (β=+0.337) raise it. Home and work meals are rated healthier than hawker center meals. Warning: random effects covariance is singular — near-zero between-person variance in healthiness after accounting for food tags.
• Binary logistic (feeling ≥ 4 vs < 4): Pseudo R²=0.108. Wholegrain (β=+1.052, p<0.001), vegetables (β=+0.479, p=0.022), and proteins (β=+0.440, p=0.017) predict high feeling. Eating alone is the strongest negative predictor (β=−0.893, p=0.001).
• Participant clustering (k=3): Silhouette = 0.157 (weak but interpretable). Cluster 0 (n=69): low feeling (3.73), low healthiness (3.08), moderate logging. Cluster 1 (n=56): high feeling (4.03), very high healthiness (4.30), low meal count (6.4) — likely sparse loggers inflating means. Cluster 2 (n=114): highest feeling (4.24), moderate healthiness (4.06), heavy loggers (29 meals). Cluster composition is independent of study arm (χ²=0.37, p=0.83).
• Interaction (healthy × socialWith → feeling): The hedonic boost from perceived healthiness is strongest when eating with a partner and weakest when eating alone. Social context modulates the health→feeling link.

A2: Hedonic Blind Spot Quantification

Blocked Data Ready Medium Effort Core

Research Question: Can VLMs predict hedonic attributes (feeling, social context, place) from meal images alone?

Method: Zero-shot VLM probing: prompt GPT-4V/Gemini/Claude with MakanMemo images, ask to predict feeling(1-5), place, social context. Compare against ground truth. Compute accuracy, confusion matrices.

Data Needed: MakanMemo images + metadata labels. VLM API access.

CHI Contribution: Quantifies the 'hedonic blind spot' of VLMs — what percentage of eating experience is invisible to vision. Motivates human-in-the-loop.

Novelty vs. Prior Work: No prior work benchmarks VLMs on hedonic/contextual meal attributes. Only food type and nutrition are benchmarked.

CHI Session fit: AI Systems for Human Goals

A3: Visual Dietary Diversity Metric

Data Ready Medium Effort Supporting

Research Question: Does embedding-space visual diversity of meals correlate with hedonic well-being?

Method: Compute per-participant meal embedding spread (CLIP/SigLIP) over time windows. Correlate with average feeling and healthy ratings. Propose a 'visual dietary diversity' index.

Data Needed: Pre-computed embeddings (have them) + metadata

CHI Contribution: Novel computational metric for dietary variety that is grounded in both visual similarity and subjective well-being.

Novelty vs. Prior Work: Dietary diversity indices exist in nutrition science but none use vision embeddings or link to hedonic data.

CHI Session fit: People and Data

B1: Singaporean Food VLM Stress Test

Blocked Data Ready Medium Effort Core

Research Question: How well do SotA VLMs recognize Singaporean food vs. Western cuisines?

Method: Benchmark 6+ VLMs on FoodSG-233 (91 Singaporean categories). Compare top-1/top-5 accuracy against published results on Western food datasets (Food-101, FoodNExTDB). Error taxonomy: wrong dish, wrong cuisine, wrong cooking style.

Data Needed: FoodSG-233 images (78K, 91 categories). VLM API access.

CHI Contribution: Extends Food-500 Cap and WorldCuisines findings with a focused Southeast Asian benchmark. Actionable error taxonomy for VLM developers.

Novelty vs. Prior Work: WorldCuisines covers breadth (2.4K dishes, 30 languages) but lacks depth on any single cuisine. FoodSG-233 is a deep single-culture benchmark.

CHI Session fit: AI Explanations and Decision Support in Healthcare

B2: kNN Label Transfer vs. VLM Zero-Shot

Partially Blocked Data Ready Low Effort Core

Research Question: Can curated cultural embeddings outperform VLM zero-shot for food labeling?

Method: kNN in CLIP/SigLIP/DINOv2 embedding space: transfer FoodSG-233 labels to MakanMemo images. Compare against VLM zero-shot classification. Measure precision, recall, cultural specificity.

Data Needed: Pre-computed embeddings for both datasets (have all 6 .npz files)

CHI Contribution: Shows that culturally curated reference data + simple retrieval can beat general-purpose VLMs. Argues for community-driven food knowledge bases.

Novelty vs. Prior Work: F4-ITS shows retrieval helps, but not specifically for cultural food transfer. No prior work compares kNN-from-cultural-dataset vs. VLM.

CHI Session fit: Human Steering and Interaction with AI

B4: Image-to-Name-to-Nutrition Pipeline Validation

Partially Blocked Partial Data Medium Effort Core

Research Question: Can we reliably map MakanMemo meal images to FoodSG-233 food names at sufficient accuracy to derive meaningful nutritional estimates?

Method: Run the full pipeline: (1) assign FoodSG-233 names to MakanMemo images via kNN label transfer and VLM zero-shot, (2) map predicted names to a Singaporean nutrition database (e.g., HPB Energy & Nutrient Composition), (3) compute estimated nutritional values. Validate against a nutritionist-annotated subset. Measure: top-1/top-5 naming accuracy, nutritional estimation error (kcal, macros) at different confidence thresholds, and the error propagation from misclassification to nutritional estimate.

Data Needed: Pre-computed embeddings (have them), FoodSG-233 labels (have them), Singaporean nutrition database (publicly available), nutritionist annotations on a validation subset (~200-500 images)

CHI Contribution: Establishes whether the image→name→nutrition pipeline is viable for Singaporean meals using existing data assets. Quantifies the accuracy threshold at which food naming becomes 'good enough' for dietary assessment — a practical prerequisite that most food-AI papers assume but do not validate.

Novelty vs. Prior Work: Most food recognition papers report classification accuracy in isolation. Few trace the error propagation through to nutritional estimation, and none do so for a Southeast Asian cuisine using culturally curated reference data.

CHI Session fit: AI Systems for Human Goals

B3: Data Sufficiency Analysis for Cultural VLM Adaptation

Partial Data High Effort Supporting

Research Question: Do we have sufficient data to fine-tune VLMs for multicultural food recognition?

Method: Learning curve experiments: fine-tune a small VLM (LLaVA/Qwen2-VL) on increasing subsets of FoodSG-233. Plot accuracy vs. data size. Identify minimum data needed per category. Compare with domain-adaptive (DAR) approaches.

Data Needed: FoodSG-233 labeled images. GPU compute for fine-tuning.

CHI Contribution: Practical guidance for teams wanting to adapt VLMs to underrepresented cuisines. Answers 'how much data is enough?'

Novelty vs. Prior Work: DAR and RAR exist but learning curves for cultural food adaptation haven't been characterized.

CHI Session fit: HCAI and Collaboration

C1: Complementarity Potential Analysis

Blocked Partial Data Medium Effort Core

Research Question: What types of VLM food recognition errors are resolvable through human clarification?

Method: Run VLMs on MakanMemo images. For each error, annotate: (a) error type (visual ambiguity, cultural gap, invisible attribute, portion confusion), (b) what human knowledge would resolve it. Compute complementarity potential (Hemmer et al. 2024).

Data Needed: VLM predictions on MakanMemo + manual annotation of errors

CHI Contribution: First systematic analysis of complementarity potential in food recognition. Maps the design space for where human input adds most value.

Novelty vs. Prior Work: Hemmer et al. define complementarity theory; this applies it to food informatics specifically.

CHI Session fit: Human-AI Decision Making

C2: Uncertainty-Resolving Food Dialogue System

Blocked Needs Build High Effort Core

Research Question: Can an 'inquisitive VLM' that asks uncertainty-resolving questions improve dietary assessment accuracy?

Method: Build prototype: VLM analyzes image → detects uncertainty (entropy/confidence) → generates targeted clarifying questions (cooking method? ingredients? cultural origin?) → user answers → VLM refines prediction. Evaluate: accuracy gain, question relevance, user burden (SUS/NASA-TLX). Wizard-of-Oz or functional prototype.

Data Needed: MakanMemo images + VLM pipeline + user study participants

CHI Contribution: System contribution: a conversational dietary assessment tool. Design patterns for uncertainty-resolving food dialogue. User study validates human-AI complementarity.

Novelty vs. Prior Work: SnappyMeal asks follow-up questions but not uncertainty-driven. FoodyTalk is conversational but doesn't use VLM vision. This combines both.

CHI Session fit: Collaborating with AI / LLM Interaction & Conversational Agents

C3: Task Allocation Framework for Meal Analysis

Blocked Partial Data Medium Effort Supporting

Research Question: How should human and VLM divide labor for comprehensive meal understanding?

Method: Define meal understanding dimensions: dish name, ingredients, cooking method, portion, cultural origin, nutritional estimate, hedonic quality. For each, measure VLM accuracy and assess human annotation cost. Plot Pareto frontier of accuracy vs. burden. Derive optimal task allocation.

Data Needed: VLM predictions on multiple dimensions + human annotations (subset)

CHI Contribution: Design framework contribution. Prescriptive guidance for building human-AI dietary assessment systems. Extends Holter & El-Assady's agency-interaction-adaptation model to food domain.

Novelty vs. Prior Work: Puerta-Beldarrain et al. (2025) propose 5 levels of human integration but not for food. No food-specific task allocation framework exists.

CHI Session fit: AI Collaboration in Practice

D1: Embedding-Based Intervention Effect Detection

Data Ready Medium Effort Supporting

Research Question: Can temporal meal embedding trajectories detect dietary behavior change during health interventions?

Method: Track per-participant meal embedding centroids over time (HAPPY/SMART trial periods). Detect shifts in embedding space using change-point detection. Correlate with self-reported healthiness trends. Compare visual trajectory analysis vs. traditional food tag analysis.

Data Needed: Longitudinal participant data with embeddings (have both)

CHI Contribution: Novel method for computationally measuring dietary intervention effectiveness from images alone. Potential tool for public health researchers.

Novelty vs. Prior Work: No prior work uses vision embedding trajectories for longitudinal dietary behavior change detection.

CHI Session fit: Self-Tracking and Personal Health Informatics

D2: Social Context and Visual Meal Signatures

Done Data Ready Low Effort Supporting

Research Question: Does social eating context produce visually distinct meal patterns?

Method: Compare embedding distributions across _with categories (Myself/Family/Friends/Partner) stratified by meal type and place. Use UMAP visualization + statistical tests (PERMANOVA on embeddings). Identify 'social meal signatures.'

Data Needed: Embeddings + metadata (have both)

CHI Contribution: Computational evidence linking social context to visual meal characteristics. Supports designing food PI tools that surface social eating patterns.

Novelty vs. Prior Work: Social eating is studied in nutrition science via surveys. No one has analyzed it through meal image embeddings.

CHI Session fit: People and Data

Results

• Analysis set: 6,024 meals with embeddings + social context across 3 models (CLIP 512d, SigLIP 1152d, DINOv2 1024d).
• PERMANOVA (global): Social context groups have statistically different centroid locations in embedding space across all three models. CLIP: F=13.576, p=0.001. SigLIP: F=17.251, p=0.001. DINOv2: F=6.345, p=0.001. Meals eaten alone look visually different from meals eaten with others.
• Stratified by place (CLIP): The social context signal persists within every eating location. Hawker Center F=3.994, Home F=4.187, Restaurant F=3.152, Work F=3.318 — all p=0.001. This is not just a place confound.
• Largest separation: Myself–Friends is the most distant pair across all models (CLIP: 0.0215, SigLIP: 0.0254, DINOv2: 0.0909). Family–Partner are most similar. People eat visually different food when alone vs. with friends.
• Within-person visual diversity: Kruskal-Wallis H=7.398, p=0.060 — borderline non-significant. Family meals show lowest within-person diversity (mean dist 0.383), Partner meals the highest (0.429). Trend suggests family meals are more visually repetitive but not statistically confirmed.
• Model agreement: CLIP and DINOv2 agree on centroid distance rankings (ρ=0.829, p=0.042). SigLIP and DINOv2 also agree (ρ=0.829, p=0.042). CLIP–SigLIP agreement is weaker (ρ=0.543, p=0.266). The social signature is robust across visual and semantic embeddings.

D3: Self-Report vs. VLM Perception Audit

Blocked Partial Data Medium Effort Supporting

Research Question: Can VLMs serve as 'nutritional perception auditors' revealing gaps between self-reported food tags and actual content?

Method: Compare participant self-reported food kind tags (7 categories) against VLM-predicted food properties. Identify systematic biases: do people over-report vegetables? Under-report deep-fried? How does this vary by feeling/healthy ratings?

Data Needed: MakanMemo images + self-reported tags + VLM predictions

CHI Contribution: Methodological contribution: using VLMs to audit self-report bias in dietary assessment. Connects to literature on dietary misreporting.

Novelty vs. Prior Work: Dietary self-report validation traditionally uses biomarkers or doubly-labeled water. Using VLMs as automated auditors is novel.

CHI Session fit: Human Behavior with AI Systems

D4: Multimodal Meal Understanding: Vision + Context Fusion

Data Ready Medium Effort Supporting

Research Question: Can multimodal fusion of meal images and experiential context outperform either alone for dietary pattern analysis?

Method: Train models: (a) image-only (embedding), (b) context-only (place, with, activity), (c) multimodal fusion. Predict meal healthiness, food type. Measure performance gap. Ablation study on which context signals help most.

Data Needed: Embeddings + all metadata fields

CHI Contribution: Demonstrates that experiential context meaningfully improves AI meal understanding. Quantifies the value of human-reported context vs. vision alone.

Novelty vs. Prior Work: DietGlance and SnappyMeal use multimodal input but don't ablate vision vs. context contributions. No one quantifies the marginal value of experiential metadata.

CHI Session fit: Personalization and Human-AI Alignment