‘Each time the distance between the lips increases, it sends out a signal so that we know when the test subject takes a bite.’ Screenshot from one of Michele Tufano’s demonstration videos. PhD student Michele Tufano is designing software to use a computer to analyse how people eat. ‘To do so, I had to do a lot of snacking on camera.’
Measuring eating behaviour is difficult. Researchers now record their subjects as they eat. Then, someone – usually master students – manually annotates all chewing and swallowing movements. A time-consuming job, Michele Tufano explains. ‘For a 12-minute video, you sometimes spend almost an hour annotating. Imagine you have a study with many subjects or multiple conditions; then you might have to analyse two hundred videos.’
Tufano is trying to automate that process. ‘To do that, I use open-source facial recognition software available online. The software puts a 3D mask over the image of the face. Such technology is also behind the popular filters of Instagram or Snapchat. We applied this software to videos collected by the Sensory Science and Eating Behaviour group.’
Mouth opening
He marked the face and the position of the lips (see also the video below). ‘Across the lips is a horizontal and a vertical line. These indicate the distance between the corners of the mouth and between the lower and upper lips. I then wrote some code that uses these two lines to register the opening of the mouth as a signal. Every time the distance between the lips increases because the mouth opens, the signal goes up, and that’s how we know when the subject takes a bite.’
To get from a signal to a bite counting tool, Tufano had to teach the computer what mouth-opening threshold belongs to a new bite. ‘Your face is constantly moving when you eat. Even with chewing, sometimes your mouth opens a little further. In this case, we only wanted to measure the bites. Every face has a slightly different shape, so a threshold value was not universal. We used three different algorithms to learn what the optimal shape is for each person. In the end, we were able to use the computer to achieve an accuracy of eighty per cent compared to manual bite counting; that is good, but should actually be even better.’
Rule-based vs deep learning
For this research paper, Tufano used a rule-based system instead of deep learning to easily adjust it to different people. ‘When trying the algorithm in a nursing home, we encountered one elderly person who had a stroke. Due to this condition, his mouth was always open, which led the program to count many extra bites, which were all false positives. Thanks to the rule-based program, we could adjust the threshold for that person to make the program work and count only true positives.
A deep learning model would increase the accuracy, but is only suitable for some different people. ‘That’s because that model is trained with reality-based data that is provided. In the nursing home example: if the training data does not include someone with a stroke, for example, they do not recognise that as true. For the next paper, we are exploring more generalised systems with deep learning to achieve better accuracy and even count chews and sips.’
Real-time
‘In addition, we want to develop the software further so that we will also be able to use it to register sips and chewing movements, and the duration of chewing, amongst others,’ says Tufano. ‘This technology allows us to generate results very quickly. We could even use it in real-time, so that you can immediately see in numbers what happens to someone’s eating behaviour if you give the subject a hard sandwich instead of a soft one, for example.’
He even dares to think about another application: ‘It also seems to be useful at our kitchen tables. We know that the faster you eat, the more you eat. If we can tell people they are eating too fast, we might also be able to make them eat more slowly and thus less. That could potentially reduce eating disorders and obesity.’
