Recommendations from generative AI tools like ChatGPT are always just a few clicks away. But should we give into temptation… or heed their creator’s warnings not to seek advice for important information?

To test this out, we pitted ChatGPT and Microsoft Copilot against each other and marked their answers against high-quality behavioural science evidence.

We found that it is essential to understand the limitations of generative AI when it comes to behaviour change advice for healthcare marketing because these tools can often get things wrong.

Pre-Match Analysis
To find a winner, we needed a focus area, so we chose a topic with ample evidence:

Smoking cessation: what helps people quit smoking

We then asked ChatGPT and Microsoft Copilot, two of the most popular generative AI tools, for the most effective influencing techniques to use to help someone quit smoking. Next, we compared their answers to the data from University College London (UCL) Centre for Behaviour Change’s specialist Smoking Cessation Prediction Tool – a specialist machine learning tool which, using the highest quality evidence available to date, predicts what percentage of a population will quit smoking based on which combination of specialist behaviour change techniques are used (Hastings et al., 2023).

So how do ChatGPT and Microsoft Copilot fair?

Three, Two, One: Fight
And we’re off – right into the action. Both tools enter the ring fighting and hit the top suggestion correctly (pharmacological support). However, we then see the tools flag behind, each missing out on several key, high-impact recommendations. ChatGPT edges ahead at the last minute due to a mis-recommendation from Copilot, but in the end, it’s UCL’s specialist Smoking Cessation Prediction Tool that emerges victorious. Let’s break this battle down in our post-match analysis.

Comparing Microsoft Copilot and ChatGPT’s answers versus the latest smoking cessation evidence


Back of the net
We’ll start with the positives. Compared with the latest evidence from UCL on what’s predicted to have the biggest impact on quit rates, both ChatGPT and Microsoft Copilot perform well, identifying five highly effective techniques, including:

  • Pharmacological support, e.g. nicotine patches and vapes
  • Social support, e.g. help from family
  • Problem-solving/action planning, e.g. help from others to foresee/plan for challenges
  • Self-monitoring, e.g. noticing triggers and adapting to life
  • Goal setting, e.g. setting a quit date

But it wasn’t all plain sailing.

Missed chances (and even an own goal)
While they started well, both tools missed out on some key techniques:

  • Missed out: Behavioural substitution, e.g. swapping smoking for a different habit
  • Missed out: Reframing, e.g. shifting from viewing smoking as relieving to causing stress
  • Missed out: Information about social impact, e.g. considering harm to others
  • Mis-recommended: Information on health risks of smoking, e.g. yellow teeth

Where tools mis-recommend is particularly worrying as, we know that simply informing smokers of how harmful smoking is ignores the role of addiction, habit, stress, social norms and other factors in driving smoking behaviour – meaning that this technique is unlikely to be effective and can even be counter-productive as most people already know smoking is harmful (Lancaster and Stead, 2017; and Beard et al., 2019).

What happened?

Three factors explain why generative AI tools just aren’t there yet with their behaviour change recommendations for healthcare marketing. These tools simply:

  1. Don’t have access to all the evidence they need to make full recommendations (such as the research held online by UCL’s Smoking Cessation Prediction Tool)
  2. Are not aimed at solving specific problems like helping people quit smoking, so they lack the appropriate frameworks to assess evidence for quality, causation and other key factors. They are also not human, so don’t understand nuances like this on a human level.
  3. Cannot reliably use, define or explain behavioural science terms, so don’t always make strong recommendations. In the experiment, both tools often recommended overall methodologies, such as Motivational Interviewing, rather than the key powerful techniques within these.

In summary: it was never a fair fight
Generative AI tools like ChatGPT and Microsoft Copilot can offer incorrect, ineffective, unclear or counterproductive advice because currently they simply are not designed with this in mind.

They are not built to assess the evidence base, synthesise the complex factors that influence human behaviour, and make robust reasoned recommendations. Specialist tools, like UCL’s Smoking Cessation Prediction Tool, however, are built with a mission in mind, the right data, and specialist support to achieve their goals and, as such, can build recommendations and predictions from the latest data.

So, while it’s tempting to turn to popular generative AI tools for quick answers, it’s crucial to remember that human expertise, supported by the latest evidence and specialist tools, is essential for designing and delivering effective behaviour change interventions for healthcare marketers.

This reflects our approach here at HRW: we always put humans first and, where AI can truly enhance our offering, we use it.

Want to learn more about the latest developments in healthcare AI?

Sign-up for our complimentary webinar series: Harnessing the Power of AI in Market Research



Beard, E., West, R., Lorencatto, F., Gardner, B., Michie, S., Owens, L., & Shahab, L. (2019). What do cost-effective health behaviour-change interventions contain? A comparison of six domains. PLOS ONE, 14(4), e0213983. https://doi.org/10.1371/journal.pone.0213983

Hastings, J., Glauer, M., West, R., Thomas, J., Wright, A., & Michie, S. (2023). Predicting outcomes of smoking cessation interventions in novel scenarios using ontology-informed, interpretable machine learning. Wellcome Open Research, 8, 503. https://doi.org/10.12688/wellcomeopenres.20012.1

Lancaster, T., & Stead, L. F. (2017). Individual behavioural counselling for smoking cessation. Cochrane Database of Systematic Reviews, 3. https://doi.org/10.1002/14651858.CD001292.pub3

Apply Now!

Get in touch:

    I would like to receive news and information from HRW
    I agree to the HRW privacy policy

      Sign up to our newsletter

      If you wish to hear about our latest blogs, podcasts, webinars, and team news, simply enter your email to sign up to our monthly newsletter.


      We use cookies to give you the best possible experience. We can also use it to analyze the behavior of users in order to continuously improve the website for you. Data Policy

      analytical Essential