A number of years back some very wild claims were being made about the future of big data. Some were suggesting that traditional market research would be obsolete within a decade. The hype seems to have died down now as people have grappled with the new digital technologies and started to work out what it works for and what it doesn’t. Generally speaking, rumours of the demise of traditional market research have been greatly exaggerated.

Broadly speaking, big data refers to the collection of massive amounts of real-life data, from what people are searching for online to their actual purchases. The logic went something like, ‘Why ask people to tell you what they think they will do, when you can collect data on what they are actually doing?’ Whilst this may have started as a merely observational exercise, increasingly, companies claim they can segment customers according to personality types and influence their behaviour with targeted messaging. Although they later downplayed their contribution, Cambridge Analytica are widely suspected to have helped influence both, the presidential campaigns of Trump and the UK Brexit referendum.

One of the frequent criticisms of big data is that it can tell you about the what, but not the why. It may give you detailed intelligence on what people are doing, but market research is more interested in the beliefs and emotional drivers that influence behaviour. Aside from being able to map trends, big data is also poor at looking into the future and exploring hypothetical situations. If a company wants to know how its new product will be perceived and will fit into the market place, and test concepts and messaging, then you need to put that messaging in front of real people and ask them.

Social media and online content monitoring is a subset of big data, focusing on content mined from popular online social media platforms and other internet resources. There are a growing list of tools available which will accept complex search terms to focus on very specific content. There are also a growing list of potential uses for this type of research.

Social media research can be used to either monitor awareness of a subject before and after marketing campaigns. Pharma companies may use it at medical conferences to evaluate response to the release of critical pivotal trial data. Increasingly they may use it to identify online influencers. Whilst traditionally Pharma has sought to partner with Key Opinion Leaders from the top flight of academic researchers, there is a new wave of digital influencers who are sometimes no more than digital savvy medical students with massive followings on Twitter. Likewise, for rare diseases like Haemophilia, sometimes a parent of a Haemophiliac child may run a blog with thousands of followers in the haemophilia space, so identification of influencers in the new digital media has become as important as building relationships with traditional academic KOLs.

Likewise, some patient support groups involve individuals that are pro-actively engaged and highly educated about their own condition. So, if one can gain access to their forums, one may discover that patients are aware of an impending drug launch in their disease and are already discussing potential pros and cons of this new agent.

Monitoring of social media is also used for PR and reputation management. I have seen this used effectively to gain early warning of the kind of negative story that can go viral within hours and cause a great deal of damage unless dealt with early.

Alternatively, some may want to look at their presence with respect to a particular therapeutic area in the online space. A large pharma company which had a very strong reputation in the fields of Growth Hormone and Haemophilia, wanted to assess how they were portraying themselves online. They frequently carried out sponsorship of events with patient organisations and support groups in both areas and were interested to see how they were being portrayed online in comparison with competitor companies. The research was also able to highlight an urgent need to tell positive stories, to counter a wall of negative online content in both conditions.

One of the main limitations with social media monitoring, centres around what can and can’t be accessed. Quite rightly, much of our content online is protected from public view and exploitation. Private Facebook content for instance, cannot be accessed by Social media search engines, although content on public Facebook Groups is fair game, which means some content from popular patient support groups in conditions like Asthma can be collected. (Whilst private Facebook content is not available to social media search engines, the degree to which Facebook sells data to private bidders lacks transparency and is under close scrutiny of regulatory bodies at the moment.) Twitter is all publicly accessible, but content is limited by the format, so is often of limited value.

Probably the least effective way to use it that I have seen, is to find out what people are saying about a company’s products. A number of years back, I was beta-testing some social media software for a company and we were doing live demos at a trade fair. A couple of executives from a German yoghurt brand were interested in what customers were saying about their brand. I set up a quick search and clicked go! Nothing came up. A lot of firms assume that somewhere out in cyberspace thousands of people are talking about their products. Much of the time that just isn’t the case. People tend not to spend a lot of time talking about products unless they are very unhappy with them. (Then you are hardly dealing with a balanced sample.) Having said that though, social media have been added as one of the channels through which Pharma companies are supposed to track Adverse events and product complaints.

Some years ago, I also helped present to the Bank of England who were curious to see if it could be used to keep tabs of economic sentiment. Current methods of monitoring economic confidence have some built in lag, which delays the Bank of England’s ability to react to rapid changes. Again, it became quickly obvious that people tend not to discuss economic confidence directly with friends online. But they do however, tell friends when they have bought a new car or booked a holiday, so we thought this may be usable as a proxy. (Unsurprisingly, it turned out the BoE had some much more powerful toys at their disposal.)

So, one of the important learnings, is that the content of social media conversations is limited to a relatively narrow range of topics. It is well understood in the market research world that people are very poor witnesses to their own behaviour. In short, they often say they will do one thing, but actually end up doing something else entirely in reality. This disconnect between how we portray ourselves and how we actually are, is even wider on social media. Sitting safely in our homes we feel comfortable projecting a version of ourselves that may be wildly at odds with reality. But there are also social norms in terms of what people discuss. Being forced to sort through vast quantities of social media content is an eye opener. Most content is depressingly banal. Market research is often interested in emotional triggers that drive decision making, but people are rarely confessional about their insecurities online. People are quite content to express outrage and anger, but rarely reveal vulnerabilities.



So, another limitation with searching social media content is that you have no control on the subjects that people talk about. Even when able to access the conversations of asthma sufferers in a public Facebook Group forum, one may have to wait an eon till somebody starts a thread about the product or brand that you are interested in hearing about. Then as a researcher you have no control over the content or direction of that conversation. That’s partially why many market research agencies have taken to setting up their own artificial social networks to populate with panels of patients with different conditions in which they can at least guide and control the conversation. They can be run like virtual focus groups that run for days and weeks at a time capturing patients’ thoughts and feelings in real time when they occur, instead of in the artificial setting of a focus group studio.

The final challenge about monitoring of social media and online content is simply how to sort and deal with the vast quantities of material. There are companies out there that claim to be able to assess sentiment of an article automatically. Frankly I’m still a sceptic. I worked with a social media search engine that had built-in qualitative coding tools, but it was very labour-intensive trawling through hundreds of pages of content looking for important themes and trying to establish whether a piece of content is positive or negative for your client. And that’s the point. Sentiment is relative and depends on where you are standing. A drug launch may be positive for the pharma company that launched it, but negative for their competitors. I haven’t yet seen evidence of an AI that is capable of making those kinds of decisions accurately. Which means it has to be done the slow (and somewhat dull) way using the human mind!

In future, undoubtedly, big data will become more prevalent and more effective. Google started by recording what we search for, then monitoring our emails, but now they have sensors in our houses and our phones which reveal our whereabouts and daily movements. If they ever work out how to process all this vast blizzard of data, they will be closer than ever to understanding what makes us tick. But in the meantime, I still feel strongly that there is no substitute for two people having a guided conversation about why they do what they do. It’s not perfect, but it’s still the fastest way we have to unlocking what people’s hopes, beliefs and fears are around any given subject.

By Glyn Griffiths
Associate Consultant

Apply Now!

Get in touch:

    I would like to receive news and information from HRW
    I agree to the HRW privacy policy

      Sign up to our newsletter

      If you wish to hear about our latest blogs, podcasts, webinars, and team news, simply enter your email to sign up to our monthly newsletter.


      We use cookies to give you the best possible experience. We can also use it to analyze the behavior of users in order to continuously improve the website for you. Data Policy

      analytical Essential