Recommendations from generative AI tools like ChatGPT are always just a few clicks away. But should we give into temptation… or heed their creator’s warnings not to seek advice for important information? To test this out, we pitted ChatGPT and Microsoft Copilot against each other and marked their answers against high-quality behavioural science evidence. We found that it is essential to understand the limitations of generative AI when it comes to behaviour change advice for healthcare marketing because these tools can often get things wrong. Pre-Match Analysis To find a winner, we needed a focus area, so we chose a topic with ample evidence: Smoking cessation: what helps people quit smoking We then asked ChatGPT and Microsoft Copilot, two of the most popular generative AI tools, for the most effective influencing techniques to use to help someone quit smoking. Next, we compared their answers to the data from University College London (UCL) Centre for Behaviour Change’s specialist Smoking Cessation Prediction Tool – a specialist machine learning tool which, using the highest quality evidence available to date, predicts what percentage of a population will quit smoking based on which combination of specialist behaviour change techniques are used (Hastings et al., 2023). So how do ChatGPT and Microsoft Copilot fair? Three, Two, One: Fight And we’re off – right into the action. Both tools enter the ring fighting and hit the top suggestion correctly (pharmacological support). However, we then see the tools flag behind, each missing out on several key, high-impact recommendations. ChatGPT edges ahead at the last minute due to a mis-recommendation from Copilot, but in the end, it’s UCL’s specialist Smoking Cessation Prediction Tool that emerges victorious. Let’s break this battle down in our post-match analysis. Comparing Microsoft Copilot and ChatGPT’s answers versus the latest smoking cessation evidence Back of the net We’ll start with the positives. Compared with the latest evidence from UCL on what’s predicted to have the biggest impact on quit rates, both ChatGPT and Microsoft Copilot perform well, identifying five highly effective techniques, including: Pharmacological support, e.g. nicotine patches and vapes Social support, e.g. help from family Problem-solving/action planning, e.g. help from others to foresee/plan for challenges Self-monitoring, e.g. noticing triggers and adapting to life Goal setting, e.g. setting a quit date But it wasn’t all plain sailing. Missed chances (and even an own goal) While they started well, both tools missed out on some key techniques: Missed out: Behavioural substitution, e.g. swapping smoking for a different habit Missed out: Reframing, e.g. shifting from viewing smoking as relieving to causing stress Missed out: Information about social impact, e.g. considering harm to others Mis-recommended: Information on health risks of smoking, e.g. yellow teeth Where tools mis-recommend is particularly worrying as, we know that simply informing smokers of how harmful smoking is ignores the role of addiction, habit, stress, social norms and other factors in driving smoking behaviour – meaning that this technique is unlikely to be effective and can even be counter-productive as most people already know smoking is harmful (Lancaster and Stead, 2017; and Beard et al., 2019). What happened? Three factors explain why generative AI tools just aren’t there yet with their behaviour change recommendations for healthcare marketing. These tools simply: Don’t have access to all the evidence they need to make full recommendations (such as the research held online by UCL’s Smoking Cessation Prediction Tool) Are not aimed at solving specific problems like helping people quit smoking, so they lack the appropriate frameworks to assess evidence for quality, causation and other key factors. They are also not human, so don’t understand nuances like this on a human level. Cannot reliably use, define or explain behavioural science terms, so don’t always make strong recommendations. In the experiment, both tools often recommended overall methodologies, such as Motivational Interviewing, rather than the key powerful techniques within these. In summary: it was never a fair fight Generative AI tools like ChatGPT and Microsoft Copilot can offer incorrect, ineffective, unclear or counterproductive advice because currently they simply are not designed with this in mind. They are not built to assess the evidence base, synthesise the complex factors that influence human behaviour, and make robust reasoned recommendations. Specialist tools, like UCL’s Smoking Cessation Prediction Tool, however, are built with a mission in mind, the right data, and specialist support to achieve their goals and, as such, can build recommendations and predictions from the latest data. So, while it’s tempting to turn to popular generative AI tools for quick answers, it’s crucial to remember that human expertise, supported by the latest evidence and specialist tools, is essential for designing and delivering effective behaviour change interventions for healthcare marketers. This reflects our approach here at HRW: we always put humans first and, where AI can truly enhance our offering, we use it. Want to learn more about the latest developments in healthcare AI? Sign-up for our complimentary webinar series: Harnessing the Power of AI in Market Research References Beard, E., West, R., Lorencatto, F., Gardner, B., Michie, S., Owens, L., & Shahab, L. (2019). What do cost-effective health behaviour-change interventions contain? A comparison of six domains. PLOS ONE, 14(4), e0213983. https://doi.org/10.1371/journal.pone.0213983 Hastings, J., Glauer, M., West, R., Thomas, J., Wright, A., & Michie, S. (2023). Predicting outcomes of smoking cessation interventions in novel scenarios using ontology-informed, interpretable machine learning. Wellcome Open Research, 8, 503. https://doi.org/10.12688/wellcomeopenres.20012.1 Lancaster, T., & Stead, L. F. (2017). Individual behavioural counselling for smoking cessation. Cochrane Database of Systematic Reviews, 3. https://doi.org/10.1002/14651858.CD001292.pub3 Apply Now!