Event report: Prediction markets reinventing pharma forecasting?

On September 5th of this year, eyeforpharma hosted a Forecasting Excellence and Prediction Markets webinar.

By Editor on Sep 24, 2007

On September 5th of this year, eyeforpharma hosted a Forecasting Excellence and Prediction Markets webinar. The panel of experts included Robin Hanson, Associate Professor of Economics at George Mason University; Carol Gebert, Accounts Manager at Thermo Fisher Scientific, Binyah Kesselly, Director of Enterprise Improvement and Process Excellence at Johnson and Johnson and eyeforpharma mathematician Joe Miles. All the experts agree that prediction marketing is an idea whose time has come for the pharma industry.

What are prediction markets?


As Joe Miles says, The opinion of the group as a whole is often better than that of the best member of the group. Prediction markets (PMs) exploit this phenomena by eliciting responses from a broad range of people, whose opinions are reflected in a single collective forecast. Miles gives the example of a time when he asked a group of people to estimate the number of candies in a jar. Guesses ranged from 400 to 40,000, but the average came out at 1,789. The real answer was 1,747. The average was far closer to the real number than any individual guess.

Let's say, for example, that you wanted to predict the outcome of the 2008 Presidential election. You could create an asset to sell on the open marketplace: a contract to pay $1 if Hillary Clinton were elected, perhaps. If people trade, then your contract develops a price which can go up or down, according to its perceived value. That price, it turns out, is a highly accurate estimate of what's likely to happen in the future (for example, a price of $0.40 suggests a 40% probability Clinton would get elected).

How do PMs compare to conventional forecasting methods?


In direct comparisons, prediction markets have proven to be as or more effective than traditional forecasting mechanisms. According to Robin Hanson, PMs have been used to forecast the outcomes of football games, orange juice futures, horse races and gas demands. In each case, PMs performed at least as well as conventional methods.

Hanson reminds us that, when judging the relative value of PMs against conventional forecasting tools, it's important to regard PMs as institutions, places where people gather to perform a specific task. We can then compare PMs to other institutions such as in-house forecasting teams, statistical analysis groups or outside consultants. Given a similar number of people in each mechanism, a similar amount of effort and the same set of inputs, how do the outputs compare?

In forecasting, says Hanson, outputs are the probabilities generated by forecasting mechanisms. These may be in the form of expected values, medians, predictions or expectations. To judge the value of the outputs, it's necessary to ask how accurate they are, how easily they can be manipulated, how often they'sre updated and how consistent they are across a broad range of topics.

The inputs are the same for all the mechanisms: in the case of institutions, the inputs are informable people. Says Hanson, these could be experts or not. They might know statistics, they might know about the topic, or they might be an amorphous crowd. The goal is to take what these people know and combine them into a total estimate.

Hanson points out a number of advantages to PMs: first, PMs are numerically precise estimates that can be updated as needed. PMs have proven to be consistent across many issues and topics and are very difficult to manipulate. They are at least as accurate as the alternatives, and the results are not distorted by the uninformed. As Hanson puts it, PMs tolerate fools. A prediction market, says Hanson, is a mechanism that will tap into the crowds when the crowds know something and tap into the experts when the experts know something.

Tapping into the wisdom of crowds

So how do prediction markets so effectively predict the future?

As in the candy example, crowds are often far smarter than individuals. Joe Miles calls this collective intelligence. Equally, groups are often far superior forecasters. Miles gives the example of ten pharma managers asked to estimate sales for a product about to be launched. The managers were from different groups: sales, finance, product and production. While the range of answers given was broad, the average of those answers was amazingly accurate. Prediction markets can harvest that collective foresight, says Miles, by eliciting underlying views.

According to Miles, the power of prediction markets is that they can uncover an exciting new perspective on uncertainties and contentious forecasts, highlight useful sources of data, reveal the aggregate opinion of a range of employees, overcome all types of bias, keep [the company] updated on changes dynamically, and improve significantly on existing forecasts.

How does pharma fit in?


Carol Gebert speaks about both benefits and potential barriers to the adoption of prediction markets by the pharma industry.

Not all predictions are created equal. To determine if a particular study has value, says Gebert, there are three dimensions to consider:

1. Is this a prediction worth making? In other words, will the study produce actionable results? If the prediction can only be generated when it's too late to change the conditions, then the answers may not be worth the effort and expense.

2. Is there a good community? Who's involved? Do they constitute a broad range of opinion?

3. Is there a good reward? For pharma, this is a particular challenge. Legal restraints make rewarding participants difficult.

In the past, some pharma PMs haven'st yielded great results, largely because some companies were unwilling to test the method with big, important questions. Pilot programs tended to be conservative, asking small, uninteresting questions, and few people were willing to participate. In order to get valuable, actionable results/predictions, it's necessary to ask the right questions.

Pharma companies often ask if PMs could be used to predict clinical success of a drug, for example, a Phase II clinical success. Gebert put the question to her three tests: definitely a prediction of clinical success would be very valuable. The potential contributors would make up a very valuable community with considerable insight. However, the rewards are legally constrained, making this an impractical option for prediction.

However, Gebert sees a very valuable place for PMs in the pharma industry, in marketing campaign planning.

Gebert asks the question, can we predict what would happen if we carry on marketing in the same old way versus changing our behavior to X? Sales teams, she says, already have a precedent of incentive-based pay, so rewarding them for participating in PMs doesn'st pose the same legal challenges. The community would consist of decision makers, consultants, sales staff, possibly even customers. And they could use PMs to answer such valuable questions as, Will sales figures change if we add/drop DTC advertising? and Will we sell more units if the price were dropped by X%?

Given the constraints on rewards given by pharma to respondents, marketing campaign planning may be the industry's most effective use of PMs.

Including PMs in pharma forecasting


Johnson & Johnson's Binyah Kesselly predicts that PMs will be a new direction for pharma forecasting.

Forecasting, despite skepticism, isn'st really the reading of tea leaves's or a roll of the dice to determine the outcome of sales and marketing activities, says Kesselly. Pharma already makes use of forecasting tools to answer questions such as whether generics will erode a drug's sales after patent protection expires or if new product sales will cannibalize existing brand share.

While traditional methods of forecasting have their uses, prediction market models have some distinct advantages. For one, PMs integrate knowledge from across an entire organization, taking advantage of multiple perspectives. PMs are ongoing and can be updated as needed. And PMs, says Kesselly, can integrate historical data across a wide variety of market conditions.

Harvesting the collective power and knowledge of the whole yields what Kesselly calls the prototypical consumer, the sweet spot toward which marketing efforts should be directed.

Prediction markets have drawbacks, of course; namely that they require some kind of reward for participants, what Kesselly calls a pay-out event or proxy. And PMs depend upon the involvement of many people from all across the information spectrum. But, as all the speakers agreed, one of the greatest benefits of a PM is increased understanding of the issue at hand.

In order to successfully utilize prediction markets, Kesselly says, an organization must have a department that is wholly dedicated to forecasting. It must ensure that senior management fully support the forecasting team, and that the organization is bold and willing to institute changes.

Prediction markets, applied correctly, can revolutionize the ways pharma does business. By democratizing the process of information-gathering, pharma companies can draw on information and opinions from a truly diverse pool of respondents. Says Kesselly, PMs's time in pharma has come, and it's going to take people with bold aspirations and people who want to make their mark in the pharmaceutical world to step forward and present this and implement it.