An intelligent analysis of food allergens through computer vision and generative models

Introduction. Food allergies represent a leading cause of adverse reactions and hospital admissions among children, with significant impact on quality of life and public health. The rapid and accurate detection of allergens in meals is therefore crucial for safety.

Materials and methods. We developed an AI-based prototype that combines YOLOv8n, a state-of-the-art object detection model trained on the Allergen30 dataset, with Gemini 2.0 Flash, an advanced generative model, to provide multimodal allergen analysis. All images were preprocessed and split into training (70%), validation (15%), and test (15%) sets, with careful class balancing.

Results. The system achieved high class-specific performance in detecting allergenic foods from real meal images, with mAP50 >90% and detailed contextual analysis via Gemini 2.0 Flash.

Discussion and conclusions. AI-assisted allergen analysis from meal images is feasible and shows promise, but does not replace ingredient disclosure or clinical precautions. Further development and real-world validation are warranted

INTRODUCTION

Food allergies affect up to 8% of children worldwide, accounting for a major share of food- related anaphylactic events and adverse reactions in the pediatric population ^1,10. The early and precise identification of potential allergens in daily meals is critical for both individual and public safety, especially for children with severe or multiple allergies ^2,11. Traditional approaches, such as careful label reading, manual inspection, or reliance on ingredient lists, are frequently insufficient due to human error, incomplete information, and the presence of hidden or cross- contact allergens ^12,13. The recent advancement of artificial intelligence (AI) and computer vision offers new solutions for automated food recognition and safety assessment ^3,14,15. Deep learning models such as YOLOv8 have demonstrated remarkable accuracy in object detection and classifying food items from images ¹⁶. However, visual recognition alone cannot identify invisible or trace allergens, nor can it account for complex preparation methods, contamination, or ingredient substitutions ¹⁷. To overcome these limitations, we present a novel prototype that integrates YOLOv8 for object detection with Gemini 2.0 Flash, a cutting-edge generative AI model capable of contextual inference from multimodal input ^5,6,18. This hybrid approach aims to provide not only robust visual classification but also inferential warnings about hidden or probable allergens, based on both detected items and external knowledge ^7,19. The objective of this study is to describe the technical development and preliminary evaluation of this AI- powered allergen analysis system, focusing on its potential utility for food-allergic children and their caregivers in home and public settings ⁸.

MATERIALS AND METHODS

The core dataset used for allergen recognition was Allergen30, a curated collection of 3,000 annotated images encompassing 30 common allergenic food categories including nuts, shellfish, dairy, eggs, and wheat. These categories were selected based on prevalence data in pediatric populations ³. Each image was manually annotated by trained reviewers to identify allergenic ingredients, with cross- checking to ensure accuracy. All images were resized to 416x416 pixels and converted to the YOLO label format. Dataset splitting into training (70%), validation (15%), and test (15%) sets was performed with a custom algorithm to ensure at least 10 images per class for training and a minimum of 3 per class for validation and test. Manual review and cleaning of labels and images were conducted to maximize data integrity ²⁰.

The YOLOv8n model was trained with the following hyperparameters: 10 epochs, batch size 32, image size 416x416 px, AdamW optimizer, initial learning rate 0.001, final learning rate 0.01, weight decay 0.0005, 3 warmup epochs, and no dropout. The training was performed on a workstation equipped with an Intel Core i9-11900KF 3.50 GHz CPU, NVIDIA RTX 4060 Ti GPU (16 GB VRAM), 32 GB RAM, and SSD storage running Windows 11. Model performance was evaluated using precision, recall, F1-score, and mAP50.

After object detection, Gemini 2.0 Flash (Google DeepMind) was used through API calls to provide contextual, generative analysis for each detected item. Gemini employs a multimodal transformer architecture to analyze detected food items in combination with metadata and prior knowledge. For each image, Gemini generates a risk report flagging likely allergens, cross-contamination risks, and preparation- related hazards. This report is grounded in the co-occurrence of food classes, known recipes, and typical allergen presence associated with the identified items ^6,22.

The user interface was built with Gradio for intuitive photo upload, instant image annotation, and generation of a detailed textual risk report ²³.

RESULTS

Evaluation of the prototype using the test set from Allergen30 demonstrated high detection accuracy across most allergen classes, with precision and recall typically above 0.85 ^3,4,16. The normalized confusion matrix (Fig. 1) revealed that the majority of classes had very low misclassification rates ^4,16. Mean average precision (mAP50) for the system exceeded 90% on the test set ^4,16,21. Visual inspection of model outputs confirmed reliable localization and identification of multiple allergenic items within composite meal images ²¹. Gemini 2.0 Flash provided a contextual narrative for each image, flagging hidden risks such as likely nut traces in desserts or probable milk in baked goods ^6,22,24. The Gradio interface enabled rapid and user-friendly image upload and report generation, with no significant usability issues in initial user tests ²³.

DISCUSSION

This study highlights the potential of combining computer vision and generative multimodal models to enhance allergen detection and risk evaluation from meal images ^6,8,18,22. YOLOv8n performed well in identifying common allergenic foods, with a solid mean average precision at 50% intersection over union (mAP50), a metric reflecting the model’s accuracy in localizing and classifying objects when predicted bounding boxes overlap at least 50% with the ground truth. In this exploratory phase, the model was trained for 10 epochs. Although limited, this training allowed for a preliminary assessment of feasibility. Increasing the number of epochs, together with early stopping to avoid overfitting, could improve performance in future iterations. Gemini 2.0 Flash complemented the pipeline with contextual and multimodal reasoning, adding interpretive depth beyond image recognition. Its transformer-based architecture leverages visual and textual prompts, allowing it to associate visual detections with potential allergenic risks even when they are not visible ^4,6,18,22. However, the system cannot identify invisible or trace allergens, such as those introduced through cross-contamination, and cannot replace full ingredient disclosure or clinical safeguards ^9,11,17,25. One limitation lies in the use of a benchmark dataset instead of real- world images, which limits generalizability. Furthermore, clinical validation in real-life settings is still lacking. Another concern involves ethical and legal risks: if the system misidentifies or fails to flag a relevant allergen, questions of liability arise, especially if the tool is integrated into consumer-facing applications. Developers must also consider informed consent, user education, and clear disclaimers to prevent misuse ^11,20,26.

Future directions should include more diverse and representative datasets, clinical testing, and integration with ingredient databases such as OpenFoodFacts to enrich model output. The fast pace of multimodal LLM development — now reaching Gemini 2.5 — suggests that such models may soon outperform task-specific detectors like YOLO, potentially consolidating detection and reasoning into unified AI systems. Despite current limitations, this experimental approach represents a promising step toward practical AI-assisted food safety tools for children and families managing food allergies ^8,9,18.

CONCLUSIONS

The presented AI-based prototype, combining YOLOv8n detection and Gemini 2.0 Flash generative inference, achieved high accuracy in allergen identification and produced detailed, contextual risk reports. While not a replacement for medical advice or ingredient transparency, it may serve as a valuable support tool for allergy management in both domestic and public environments. Ongoing development and real-world validation will be essential for safe and effective deployment ^8,9,18,22.

Acknowledgements

We gratefully acknowledge the authors of the Allergen30 dataset for providing open access to their data, which was utilized in this study.

Ethical consideration

As this study is based on publicly available online datasets, ethical approval and informed consent are not applicable.

Funding

This research received no external funding .

Conflicts of interest statement

The authors have no competing interests to declare.

Authors’ contributions

Methodology, G.M.; writing-original draft preparation, A.P.; data curation, A.S.C.; writing-review and editing, S.C.; supervision, M.M., V.F.; supervision, C.I.; project administration, M.M.d.G. All authors have read and agreed to the published version of the manuscript

History

Received: May 26, 2025

Published: July 28, 2025

Figures and tables

FIGURE 1. Normalized confusion matrix showing class-specific detection performance of the YOLOv8n model on the Allergen30 test set. The matrix highlights high accuracy and low rates of misclassification for the majority of allergenic food classes analyzed.