Emotion Recognition on Food Images


The relationship between emotion and food has been studied for years. On the one hand, the emotional state has an effect on our eating behaviors, including food choice and food intake. On the other hand, what we eat influence emotions. For example, energy-dense food such as sugar and fat evokes positive affects, while tasting bitter compounds evokes negative affective responses [1]. In this project, we aim to recognize underlying emotions of the food images that people have uploaded to online social networking/media sites. By training models with extracted food image features, we hope to predict the emotional responses that people have after eating these foods, and investigate the correlation between certain features (e.g., colors) and emotion.


The general procedures are as follows:

  1. Collecting training data from online social networking/media sites (Yelp, a restaurant review sites, as the current choice), including food images, and related comments or overall ratings.
  2. Extracting features from images, which can be obtained in two ways:
    • Hand-crafted low level features, such as color histograms, texture, shape, content, etc.;
    • High-level features from Convolutional Neural Networks (CNN).
  3. Getting corresponding ground truth labels for training images. It can be obtained according to a restaurant overall rating, or comments from a reviewer. Currently we only take positive and negative as two emotion labels into consideration.
  4. Training classifiers that take image features as input, and output predicted emotion label.


[1] Gibson E L. Emotional influences on food choice: sensory, physiological and psychological pathways[J]. Physiology & behavior, 2006, 89(1): 53-61.


Emotion recognition, image processing, food images,




Jiayu Shu and Pearl Pu