Prediction markets: Don’t ask people about cricket when they have no idea15.03.2018
When forecasting future demand or making accurate assumptions about how a product or concept will perform, an emerging technique utilising a live betting marketplace with dynamic pricing adjustment called ‘prediction markets’ has shown some promise. The HRW team ran a self-funded study to put this approach through its paces and found mixed results.
Forecasting how a product or concept will perform is never easy. And trying to assess future behaviour in market research is confounded by three factors:
- We don’t actually know what we would do – behavioural science has shown that we, as humans, are bad at ‘prospection’ (knowing how we would think and behave in the future).
- We are missing the social dimension – we are very influenced by the perceptions and behaviours of our peers (the herd instinct).
- The stakes aren’t high – making decisions in real life (especially health decisions for patients and physicians) is important and risky, but responding to survey questions carries less importance and deliberation.
At HRW, we are always on the lookout for ways to improve our research and overcome limitations in survey design to get us closer to the truth. We heard about an approach born from experimental economics; based in the principles of stock markets and political betting (before the era of scientific polling), which had shown use in the consumer space. It was then only natural for us as avid researchers to merit this a topic worth further investigation, to see whether this approach delivered on its promise and could be applied in healthcare research.
Prediction markets is an approach utilising a simulated betting environment. Each participant is given the same amount of virtual ‘currency’ and can place multiple bets (spread betting), or solely on one option (all in). Like a betting environment, options are shown with their potential virtual ‘payoffs’ tied to the speculated outcome of future events (success of a product or concept). Like a real market place, concepts that are more popular within the sample go up in price to reflect the demand, so become costlier to purchase (but signal to the bettor that it is a more likely bet/outcome).
In our self-funded comparative study, we worked in partnership with M3 global to construct a survey aimed at predicting the outcomes of public events. Within the survey interface we created a prediction markets approach (where people could make multiple bets on multiple options), and, for comparison, we also included a traditional (single choice) approach as a control. Subjects covered in the survey included cricket world cup, rugby world cup, the Grammy awards, and the Oscars. All participants answered all questions with both approaches so we could compare the results.
So, what did we find?
Both the traditional and prediction markets approaches were moderately effective at predicting outcomes. As you might expect, when individual responses are compared between prediction markets and control within the same survey population, the results of both approaches were well aligned with one another. However, surprisingly, in the majority of cases, the traditional approach was ever so slightly more effective than the prediction markets approach.
Furthermore, we uncovered some important limitations to this new approach:
- Participant instructions and understanding (in the pilot test) – many respondents in the initial pilot phase fed back that the prediction market question approach was not clear enough to them. Therefore, possibility for erroneous data entry was then high at this stage, and indeed many respondents did not realise that they could spread their bets.
- Not longitudinal enough – the time between fielding the survey and events occurring was very short; increasing this time gap and allowing for participants to revise their bets and sell their ‘stake’ in different options at different time points leading up to the event itself would have provided richer data to analyse.
- Difficulty of predicting ‘tournament’ outcomes – as is the nature (and also the attraction) of many tournaments, anything can happen. The best calculations cannot account for an unlikely upset and so we saw that these categories of event performed more poorly than those based on third party decision-making (voting), such as awards and elections.
- Too many choices – when presented with more potential options, respondents were more likely to spread their bets around. This finding is consistent with findings in cognitive science, that suggest ‘naïve diversification’ is at play. This was clearly evident with the Cricket World Cup predictions where there were many teams involved, and it was our observation that greater accuracy would likely be found in smaller choice sets.
In conclusion, despite predicting events no better than traditional approaches, with some refinement and careful application, prediction markets may still have a place within the market researcher’s arsenal of methodologies for one key reason: it’s far more engaging! Respondent fatigue and boredom are issues we are increasingly being faced with in market research and this approach delivers variety and social feedback that engages participants by its nature. Furthermore, using gamification to tie this virtual currency with virtual rewards or rankings (within an online community, say) we could increase the stakes and encourage participants in surveys to be more considerate about their responses. So, while it might not be well-suited to deal with the complexity that we need for it to replace our forecasting approach in cases where we are testing appeal of new devices, refinement of a set of campaign concepts or patient profile prioritisation, the prediction markets approach could add a perfect element of additional insight.
By Jaz Gill