-
Are Vision-Language Models Ready for Dietary Assessment? Exploring the Next Frontier in AI-Powered Food Image Recognition
Authors:
Sergio Romero-Tapiador,
Ruben Tolosana,
Blanca Lacruz-Pleguezuelos,
Laura Judith Marcos Zambrano,
Guadalupe X. Bazán,
Isabel Espinosa-Salinas,
Julian Fierrez,
Javier Ortega-Garcia,
Enrique Carrillo de Santa Pau,
Aythami Morales
Abstract:
Automatic dietary assessment based on food images remains a challenge, requiring precise food detection, segmentation, and classification. Vision-Language Models (VLMs) offer new possibilities by integrating visual and textual reasoning. In this study, we evaluate six state-of-the-art VLMs (ChatGPT, Gemini, Claude, Moondream, DeepSeek, and LLaVA), analyzing their capabilities in food recognition a…
▽ More
Automatic dietary assessment based on food images remains a challenge, requiring precise food detection, segmentation, and classification. Vision-Language Models (VLMs) offer new possibilities by integrating visual and textual reasoning. In this study, we evaluate six state-of-the-art VLMs (ChatGPT, Gemini, Claude, Moondream, DeepSeek, and LLaVA), analyzing their capabilities in food recognition at different levels. For the experimental framework, we introduce the FoodNExTDB, a unique food image database that contains 9,263 expert-labeled images across 10 categories (e.g., "protein source"), 62 subcategories (e.g., "poultry"), and 9 cooking styles (e.g., "grilled"). In total, FoodNExTDB includes 50k nutritional labels generated by seven experts who manually annotated all images in the database. Also, we propose a novel evaluation metric, Expert-Weighted Recall (EWR), that accounts for the inter-annotator variability. Results show that closed-source models outperform open-source ones, achieving over 90% EWR in recognizing food products in images containing a single product. Despite their potential, current VLMs face challenges in fine-grained food recognition, particularly in distinguishing subtle differences in cooking styles and visually similar food items, which limits their reliability for automatic dietary assessment. The FoodNExTDB database is publicly available at https://github.com/AI4Food/FoodNExtDB.
△ Less
Submitted 9 April, 2025;
originally announced April 2025.
-
Personalized Weight Loss Management through Wearable Devices and Artificial Intelligence
Authors:
Sergio Romero-Tapiador,
Ruben Tolosana,
Aythami Morales,
Blanca Lacruz-Pleguezuelos,
Sofia Bosch Pastor,
Laura Judith Marcos-Zambrano,
Guadalupe X. Bazán,
Gala Freixer,
Ruben Vera-Rodriguez,
Julian Fierrez,
Javier Ortega-Garcia,
Isabel Espinosa-Salinas,
Enrique Carrillo de Santa Pau
Abstract:
Early detection of chronic and Non-Communicable Diseases (NCDs) is crucial for effective treatment during the initial stages. This study explores the application of wearable devices and Artificial Intelligence (AI) in order to predict weight loss changes in overweight and obese individuals. Using wearable data from a 1-month trial involving around 100 subjects from the AI4FoodDB database, includin…
▽ More
Early detection of chronic and Non-Communicable Diseases (NCDs) is crucial for effective treatment during the initial stages. This study explores the application of wearable devices and Artificial Intelligence (AI) in order to predict weight loss changes in overweight and obese individuals. Using wearable data from a 1-month trial involving around 100 subjects from the AI4FoodDB database, including biomarkers, vital signs, and behavioral data, we identify key differences between those achieving weight loss (>= 2% of their initial weight) and those who do not. Feature selection techniques and classification algorithms reveal promising results, with the Gradient Boosting classifier achieving 84.44% Area Under the Curve (AUC). The integration of multiple data sources (e.g., vital signs, physical and sleep activity, etc.) enhances performance, suggesting the potential of wearable devices and AI in personalized healthcare.
△ Less
Submitted 13 September, 2024;
originally announced September 2024.
-
AI4Food-NutritionFW: A Novel Framework for the Automatic Synthesis and Analysis of Eating Behaviours
Authors:
Sergio Romero-Tapiador,
Ruben Tolosana,
Aythami Morales,
Isabel Espinosa-Salinas,
Gala Freixer,
Julian Fierrez,
Ruben Vera-Rodriguez,
Enrique Carrillo de Santa Pau,
Ana Ramírez de Molina,
Javier Ortega-Garcia
Abstract:
Nowadays millions of images are shared on social media and web platforms. In particular, many of them are food images taken from a smartphone over time, providing information related to the individual's diet. On the other hand, eating behaviours are directly related to some of the most prevalent diseases in the world. Exploiting recent advances in image processing and Artificial Intelligence (AI),…
▽ More
Nowadays millions of images are shared on social media and web platforms. In particular, many of them are food images taken from a smartphone over time, providing information related to the individual's diet. On the other hand, eating behaviours are directly related to some of the most prevalent diseases in the world. Exploiting recent advances in image processing and Artificial Intelligence (AI), this scenario represents an excellent opportunity to: i) create new methods that analyse the individuals' health from what they eat, and ii) develop personalised recommendations to improve nutrition and diet under specific circumstances (e.g., obesity or COVID). Having tunable tools for creating food image datasets that facilitate research in both lines is very much needed.
This paper proposes AI4Food-NutritionFW, a framework for the creation of food image datasets according to configurable eating behaviours. AI4Food-NutritionFW simulates a user-friendly and widespread scenario where images are taken using a smartphone. In addition to the framework, we also provide and describe a unique food image dataset that includes 4,800 different weekly eating behaviours from 15 different profiles and 1,200 subjects. Specifically, we consider profiles that comply with actual lifestyles from healthy eating behaviours (according to established knowledge), variable profiles (e.g., eating out, holidays), to unhealthy ones (e.g., excess of fast food or sweets). Finally, we automatically evaluate a healthy index of the subject's eating behaviours using multidimensional metrics based on guidelines for healthy diets proposed by international organisations, achieving promising results (99.53% and 99.60% accuracy and sensitivity, respectively). We also release to the research community a software implementation of our proposed AI4Food-NutritionFW and the mentioned food image dataset created with it.
△ Less
Submitted 12 September, 2023;
originally announced September 2023.
-
Leveraging Automatic Personalised Nutrition: Food Image Recognition Benchmark and Dataset based on Nutrition Taxonomy
Authors:
Sergio Romero-Tapiador,
Ruben Tolosana,
Aythami Morales,
Julian Fierrez,
Ruben Vera-Rodriguez,
Isabel Espinosa-Salinas,
Gala Freixer,
Enrique Carrillo de Santa Pau,
Ana Ramírez de Molina,
Javier Ortega-Garcia
Abstract:
Maintaining a healthy lifestyle has become increasingly challenging in today's sedentary society marked by poor eating habits. To address this issue, both national and international organisations have made numerous efforts to promote healthier diets and increased physical activity. However, implementing these recommendations in daily life can be difficult, as they are often generic and not tailore…
▽ More
Maintaining a healthy lifestyle has become increasingly challenging in today's sedentary society marked by poor eating habits. To address this issue, both national and international organisations have made numerous efforts to promote healthier diets and increased physical activity. However, implementing these recommendations in daily life can be difficult, as they are often generic and not tailored to individuals. This study presents the AI4Food-NutritionDB database, the first nutrition database that incorporates food images and a nutrition taxonomy based on recommendations by national and international health authorities. The database offers a multi-level categorisation, comprising 6 nutritional levels, 19 main categories (e.g., "Meat"), 73 subcategories (e.g., "White Meat"), and 893 specific food products (e.g., "Chicken"). The AI4Food-NutritionDB opens the doors to new food computing approaches in terms of food intake frequency, quality, and categorisation. Also, we present a standardised experimental protocol and benchmark including three tasks based on the nutrition taxonomy (i.e., category, subcategory, and final product recognition). These resources are available to the research community, including our deep learning models trained on AI4Food-NutritionDB, which can serve as pre-trained models, achieving accurate recognition results for challenging food image databases.
△ Less
Submitted 19 April, 2024; v1 submitted 14 November, 2022;
originally announced November 2022.
