MakanMemo x CHI: Research Questions & Experiments Forecast
14 experiments across 4 themes, designed for a Human-LLM/VLM diet intervention paper targeting CHI.
Built on 6,166 real-world meal photos from 249 Singaporean participants with hedonic metadata.
Dataset Profile
Eating Context Distribution
Hedonic Feeling Ratings (1–5)
Social Company
Food Kind Tags (multi-label)
Research Themes & Experiments
Experiment Prioritization Matrix
| ID | Experiment | Data? | Effort | VLM Access | CHI Fit | Anchor? |
|---|
Suggested Paper Configurations
Current food PI tools and VLMs are blind to hedonic experience. We quantify what they miss using 6K+ real meals with hedonic metadata, show VLMs cannot predict feeling/context, and demonstrate that fusing experiential context with vision improves meal understanding.
VLMs systematically fail on Singaporean food. We benchmark this gap, show that curated cultural data + kNN outperforms zero-shot VLMs, map where humans can fill remaining gaps, and use VLMs to audit self-reporting bias.
Design and evaluate a complete human-VLM collaborative dietary assessment system. Identify VLM blind spots (hedonic + cultural), characterize complementarity potential, build an uncertainty-resolving dialogue system, and derive a task allocation framework.
Comprehensive empirical foundation showing what data, methods, and human-AI configurations are needed for culturally-aware dietary intervention systems. Establishes baseline capabilities and identifies research directions.
Key Related Work Positioning
| Paper | What they do | What we add |
|---|---|---|
| WorldCuisines (Winata, NAACL 2025) | Massive multilingual food VQA (2.4K dishes, 30 languages) | Deep single-culture analysis (91 SG categories). Real meal images, not web-scraped. Hedonic metadata. |
| SnappyMeal (2025) | Multimodal AI food logging with follow-up questions, 3-week study | Uncertainty-driven questions (not generic). Cultural food focus. Larger longitudinal dataset. Hedonic context. |
| FoodyTalk (Silva, CHI 2025) | Conversational food journaling for empathy/accountability | VLM vision + conversation (not text-only). Quantitative analysis of what conversation resolves. |
| Speaking of Food (CHI 2026) | How people verbalize food experiences (temporal, associative, evaluative) | We have the quantitative counterpart: 6K meals with rated hedonic dimensions. |
| FoodCHA (Lee, 2026) | Hierarchical VLM agent for food taxonomy classification | Beyond taxonomy: hedonic, cultural context, human-in-the-loop correction. Real deployment data. |
| Hemmer et al. (2024) | Theory of human-AI complementarity in decision-making | Apply complementarity framework to food recognition specifically. Empirical validation in cultural food domain. |
The hedonic feeling and self-perceived healthiness distributions are heavily right-skewed: 83.6% of feeling ratings are 4 or 5, and 58.9% of healthiness ratings are 4 or 5. This is a well-documented ceiling effect in self-report data — participants default to high ratings to minimize cognitive effort, especially in longitudinal studies where reporting fatigue accumulates.
Contrast this with the eating context variables (place, company, meal type) which show far more uniform distributions and higher variability. The difference is telling: categorical selectors (pick from a list) impose minimal cognitive burden, while Likert-scale hedonic judgments require introspective effort that participants often shortcut.
Implication: any analysis using feeling or healthiness as a dependent variable must account for restricted range. Consider treating ratings as ordinal, collapsing to low (1–3) vs. high (4–5), or modeling ceiling effects explicitly. Results showing differences despite this compression are more robust, not less.
The dataset contains 6,166 meal observations from 249 participants. These are not independent observations — meals are nested within persons, and within-person autocorrelation is expected (dietary habits are stable). Effective sample size for between-person analyses is N ≈ 249, not 6,166.
Key considerations for statistical power:
- Participant contribution is likely unequal: some participants may have logged 100+ meals while others contributed <10. The distribution of observations per participant (and the identifiability of per-person patterns) needs characterization before any per-person analysis.
- Mixed-effects models are required for any regression — random intercepts/slopes per participant to account for non-independence.
- Temporal autocorrelation: consecutive meals from the same person are not independent. HAPPY and SMART trial arms may also introduce clustering.
- Subgroup power drops fast: splitting by place × company × meal type can produce cells with very few observations. A 4 × 4 × 4 cross yields 64 cells across 249 people — many will be sparse or empty.
- Trial arm metadata: participant-level trial assignment (HAPPY vs. SMART), demographics, and intervention period markers are needed to separate intervention effects from secular trends. This data exists but is not yet integrated into the analysis-ready CSV.
Bottom line: report per-participant observation distributions before any analysis. Effect sizes detected at the observation level (N=6K) that vanish at the participant level (N=249) are likely artifacts of within-person repetition, not real population effects.
The current analysis-ready CSV represents participants only by study arm (HAPPY vs. SMART). Richer participant-level variables exist in the source trial data but are not yet integrated:
- Day of intervention / study phase: where each meal falls in the participant's intervention timeline (baseline, active intervention, follow-up). Without this, temporal analyses (D1) cannot separate intervention effects from secular trends or seasonality.
- Study protocol details: intervention type, dosage, duration, and arm-specific design differences between HAPPY and SMART that could confound cross-arm comparisons.
- Pre-study questionnaires: baseline dietary habits, health status, BMI, dietary goals, food literacy, and other intake collected at enrollment. These are critical covariates for any between-participant analysis — without them, observed differences may reflect pre-existing traits rather than intervention or contextual effects.
- Demographics: age, gender, ethnicity, household composition — essential for assessing whether findings generalize beyond the study sample and for stratified analyses of cultural food patterns.
- Other longitudinal instruments: repeated questionnaires (mood, stress, well-being scales) collected during the trial that could serve as richer outcome variables than the single-item feeling/healthy ratings.
Action required: merge participant-level metadata from the original trial databases before running between-person analyses (A1, A3, D1, D2, D4). Within-person analyses (temporal trends, embedding trajectories) are less affected but still benefit from knowing intervention phase.
References
From the Consensus literature review and assorted papers informing this forecast.
VLM Food Recognition & Benchmarking
- Romero-Tapiador, S., Tolosana, R., Lacruz-Pleguezuelos, B., et al. (2025). Are Vision-Language Models Ready for Dietary Assessment? Exploring the Next Frontier in AI-Powered Food Image Recognition. 2025 IEEE/CVF CVPRW, 430–439. doi:10.1109/cvprw67362.2025.00047
- Ma, Z., Pan, M., Wu, W., Cheng, K. L., Zhang, J., Huang, S., & Chen, J. (2023). Food-500 Cap: A Fine-Grained Food Caption Benchmark for Evaluating Vision-Language Models. Proc. 31st ACM Intl. Conf. Multimedia. doi:10.1145/3581783.3611994
- Winata, G. I., Hudi, F., Irawan, P. A., et al. (2025). WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisines. Proc. NAACL 2025, 3242–3264. arXiv:2410.12705
- Nayak, S., et al. (2024). Benchmarking Vision Language Models for Cultural Understanding. EMNLP 2024. arXiv:2407.10920
- Wang, S., Sheng, G., Yan, H., Min, W., & Jiang, S. (2026). A comparative study of vision–language models for food ingredient recognition and nutrient estimation. Current Research in Food Science, 12. doi:10.1016/j.crfs.2026.101405
- Lee, W., Mekkoth, P., Tian, Y., Gungor, O., & Rosing, T. (2026). FoodCHA: Multi-Modal LLM Agent for Fine-Grained Food Analysis. arXiv:2605.05499
- Ma, D., Xu, Z., Xing, T., et al. (2025). Improving Food Recognition with Retrieval-Augmented and Domain-Adaptive LVLMs. ICASSP 2025. doi:10.1109/icassp49660.2025.10888217
- Asokan, R. (2025). F4-ITS: Fine-grained Feature Fusion for Food Image-Text Search. ArXiv. doi:10.48550/arxiv.2508.17037
- Zhang, J., Huang, J., Jin, S., & Lu, S. (2023). Vision-Language Models for Vision Tasks: A Survey. IEEE Trans. PAMI, 46, 5625–5644. doi:10.1109/tpami.2024.3369699
- Yin, Y., Qi, H., Zhu, B., Chen, J., Jiang, Y.-G., & Ngo, C. (2023). FoodLMM: A Versatile Food Assistant Using Large Multi-Modal Model. IEEE Trans. Multimedia. doi:10.1109/tmm.2025.3590924
Human-AI Collaboration & Complementarity
- Hemmer, P., Schemmer, M., Kuhl, N., Vossing, M., & Satzger, G. (2024). Complementarity in human-AI collaboration: concept, sources, and evidence. European Journal of Information Systems, 34, 979–1002. doi:10.1080/0960085x.2025.2475962
- Vössing, M., Kühl, N., Lind, M., & Satzger, G. (2022). Designing Transparency for Effective Human-AI Collaboration. Information Systems Frontiers, 24, 877–895. doi:10.1007/s10796-022-10284-3
- Senoner, J., Schallmoser, S., Kratzwald, B., Feuerriegel, S., & Netland, T. (2024). Explainable AI improves task performance in human–AI collaboration. Scientific Reports. doi:10.1038/s41598-024-82501-9
- Holter, S. & El-Assady, M. (2024). Deconstructing Human-AI Collaboration: Agency, Interaction, and Adaptation. Computer Graphics Forum. doi:10.1111/cgf.15107
- Puerta-Beldarrain, M., Gómez-Carmona, O., et al. (2025). A Multifaceted Vision of the Human-AI Collaboration: A Comprehensive Review. IEEE Access. doi:10.1109/access.2025.3536095
- Berretta, S., Tausch, A., Ontrup, G., et al. (2023). Defining human-AI teaming the human-centered way. Frontiers in AI. doi:10.3389/frai.2023.1250725
- Schmutz, J., Outland, N., Kerstan, S., et al. (2024). AI-teaming: Redefining collaboration in the digital era. Current Opinion in Psychology. doi:10.1016/j.copsyc.2024.101837
Conversational Agents, Food Journaling & Personal Informatics
- Silva, L. M., Lu, X., Liang, E., & Epstein, D. A. (2025). Foody Talk: Exploring Opportunities for Conversational Food Journaling. Proc. CHI 2025. doi:10.1145/3706598.3713795
- Chopra, S., et al. (2025). Engagements with Generative AI and Personal Health Informatics. Proc. ACM IMWUT, 9(3). doi:10.1145/3734508
- Shin, M., Jang, M., Cho, M., & Ryu, J.-K. (2023). Uncertainty-Resolving Questions for Social Robots. Companion HRI 2023. doi:10.1145/3568294.3580077
- Schöbel, S., Schmitt, A., Benner, D., et al. (2023). Charting the Evolution and Future of Conversational Agents. Information Systems Frontiers, 26, 729–754. doi:10.1007/s10796-023-10375-9
- Mindlin, D., Beer, F., Sieger, L. N., et al. (2025). Beyond one-shot explanations: a systematic literature review of dialogue-based xAI approaches. Artificial Intelligence Review. doi:10.1007/s10462-024-11007-7
- Li, G., Hammoud, H., Itani, H., Khizbullin, D., & Ghanem, B. (2023). CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society. NeurIPS 36. doi:10.52202/075280-2264
- Gayler, T., Sas, C., & Kalnikaité, V. (2022). Exploring the Design Space for Human-Food-Technology Interaction. ACM Trans. CHI. doi:10.1145/3484439
AI in Nutrition & Dietary Assessment
- Panayotova, G. (2025). Artificial Intelligence in Nutrition and Dietetics: A Comprehensive Review. Healthcare, 13. doi:10.3390/healthcare13202579
- Ma, P., et al. (2024). Large language models in food science: innovations, applications, and future. Trends in Food Science & Technology.
- Yang, Z., et al. (2024). ChatDiet: Empowering Personalized Nutrition-Oriented Food Recommender Chatbots through an LLM-Augmented Framework. Smart Health.
- Peihua, M., Wu, Y., Yu, N., et al. (2024). Integrating Vision-Language Models for Accelerated High-Throughput Nutrition Screening. Advanced Science, 11. doi:10.1002/advs.202403578
- Papastratis, I., Konstantinidis, D., Daras, P., & Dimitropoulos, K. (2024). AI nutrition recommendation using a deep generative model and ChatGPT. Scientific Reports. doi:10.1038/s41598-024-65438-x
- Shonkoff, E. T., Cara, K., Pei, X., et al. (2023). AI-based digital image dietary assessment methods compared to humans and ground truth. Annals of Medicine. doi:10.1080/07853890.2023.2273497
- Soekamto, Y., Lim, A., et al. (2025). Pic2Plate: A Vision-Language and Retrieval-Augmented Framework for Personalized Recipe Recommendations. Sensors. doi:10.3390/s25020449
- Hu, Y. & Zhuang, G. (2025). MultiFoodChat: A potential new paradigm for intelligent food quality inspection. ArXiv. doi:10.48550/arxiv.2510.13889
Participatory ML & Community-Engaged Approaches
- Asabor, E., Aneni, K., Weerakoon, S., & Opara, I. (2024). Applying a Community-Engaged Participatory Machine Learning Model. American Journal of Community Psychology, 74, 262–268. doi:10.1002/ajcp.12765
- Prabhakaran, V. & Martin, D. (2020). Participatory Machine Learning Using Community-Based System Dynamics. Health and Human Rights.
Additional CHI 2026 References
- CHI 2026. Speaking of Food: Understanding How People Talk About Food Experiences. Proc. CHI 2026. doi:10.1145/3772318.3791474
- CHI 2026. Envisioning Future Food, Technology, and Health: Teens' Perspectives Through Design Fiction. Proc. CHI 2026. doi:10.1145/3772318.3790644
- SnappyMeal: Design and Longitudinal Evaluation of a Multimodal AI Food Logging Application. arXiv:2511.03907
- DietGlance: Dietary Monitoring and Personalized Analysis at a Glance with Knowledge-Empowered AI Assistant. arXiv:2502.01317