Photo: Shutterstock ‘I expected ChatGPT to score better than dieticians occasionally, but I had not expected it would be this convincing’, says Daniel Kirk, a researcher of Global Nutrition. Kirk tested whether dieticians or ChatGPT were better at answering frequently asked questions on nutrition. The outcome was surprisingly clear.
Chatbot ChatGPT has been in existence for about a year and has become an integral part of some people’s daily activities. But some also fear that it could take over the jobs of individuals or even entire sectors. Daniel Kirk was curious about the effect on dieticians. ‘Is ChatGPT equally good at answering questions about nutrition?’
Convincing
Without revealing the precise nature of his investigation, Kirk asked dieticians to provide him with a number of frequently asked general questions on nutrition and the answers they would give their clients. This resulted in eight questions, which he then submitted to ChatGPT.
He then asked different dieticians and nutrition experts -to whom he also did not reveal his research topic – to assess the answers’ scientific validity, usefulness and comprehensibility. He then compared these scores for each of the criteria with the answers provided by the dieticians. He then merged the scores to arrive at a single average for ChatGPT and for the dieticians.
The average score achieved by ChatGPT was demonstrably higher than that of the dieticians for five of the questions. The scores were more or less equal for two of the questions, and for one question, the difference was minimal but slightly in favour of the chatbot. ‘In terms of the separate scores for scientific validity, usefulness and comprehensibility, ChatGPT did better than the dieticians’, says Kirk. More poignantly, statistically speaking, the dieticians did not score significantly higher than the chatbot for any of the criteria in any of the questions.
‘That surprises me. I expected the dieticians to win for at least some of the questions since they provided the questions and thus had an advantage over ChatGPT.’
Obsolete profession?
‘Critics will say that data used by ChatGPT is, by definition, dated, and that is largely correct. We used ChatGPT 3.0, the free version with input updated until September 2021. Hence, the study does not offer any ground-breaking or new insights, but that is not the goal’, Kirk clarifies. ‘All knowledge we gain in nutrition research builds on existing knowledge, so general recommendations will largely remain the same unless a revolution takes place in the field of nutrition and we discover that everything we always knew about carbohydrates, for example, is false.’
Dieticians who fear for the future of their profession need not fret: the chatbot is currently not able to replace dieticians. ‘Chatbots have no emotions, for example. Part of the dietician’s job is to provide emotional support, listen to a client, understand their perspective and then provide a personalised recommendation. In that respect, chatbots are no match for experienced dieticians. We were mainly interested in testing how good or bad a chatbot is in providing general nutritional information. Quite good, it appears, equal to or better than a dietician. We may be able to specialise this in the future.’
Future
Kirk believes that ChatGPT can add to the consultations provided by dieticians. ‘ChatGPT does not know what products exactly you should eat. But it can provide answers to general questions about nutrition, and that could prove useful if someone wants to have a snack at nine in the evening. A dietician is probably not available at that time, but a chatbot is always there.’
Kirk used a general chatbot in his research. ‘A next step might be to train a chatbot with knowledge about nutrition and include it in an app for consumers or to link it to an app consumers already use, such as MyFitnessPal. The advantage is that such advice is a lot cheaper than a consultation with a dietician. Moreover, it is practical and available instantly every hour of the day or night.’
