Understanding uncertainty and risk in pharma forecasting
Alec Finney, principal at Rivershill Consulting, on best practice in pharma forecasting
The financial meltdown of 2007/8 led to a surge in the application of risk management tools not only in the most affected financial institutions but also in every aspect of investment assessment.
Pharma invests heavily in new ideas, so it is not surprising to find more rigor in risk management in our industry.
The ways in which forecasts are created and communicated to decision makers has also changed during this time.
Forecasts now need to give some context to the single line of numbers, some expression of the certainty (or confidence) the forecaster has in the numbers.
Here are some suggested guidelines on making that happen.
The forecaster's accountabilities
The forecaster still has to deliver the most likely forecast' based on a robust set of assumptions and using good proven models.
Context to this one line forecast can be provided in two ways.
The first is to deliver an envelope of possibility,' in effect a confidence interval that displays 90% of all possible outcomes, still based on the initial set of assumptions.
How we generate this band is addressed below in the Monte Carlo simulations section.
The second way of providing context is to describe a scenario-another significant future based on a change in a small number (usually one) of key assumptions.
Language and communication
Uncertainty' and risk' are often used interchangeably, but they are not the same thing.
We can measure uncertainty and then try to manage the underlying risk.
This needs to be understood by all forecast stakeholders.
For example, the technical success of a clinical trial is rated at 80%, an expression of uncertainty of an outcome.
The cost of the trial might be $250 million, and it is this investment that is at risk.
Also, probability numbers' and probability descriptions' are not consistent.
Many studies show that, for example, probable' is judged to be between 30% and 80%; unlikely' is between 10% and 40%.
Hence, there can be a possible overlap in expectation between probable' and unlikely'.
Finally, there is a temptation to give the decision-maker as much information as possible rather than as much information as needed.
A prime example here is the scenario for all seasons', a PowerPoint chart that looks like a scene from a firework display.
Risk management tools
All the techniques below have utility in helping determine the level of uncertainty in a forecast, but they have limitations and have to be used carefully.
Monte Carlo simulations Senior managers don't generally understand Monte Carlo simulations, which is a way of combining uncertainties.
The best piece of insightful information is the Tornado Chart, where the biggest uncertainties are identified.
Plans can then be made to reduce that uncertainty (at a cost) and hence reduce the risk-better conjoint, better price access studies, better epi data, for example.
There is one significant danger in the use of Monte Carlo and that is recognizing when to use the dependency tool.
Some assumptions are linked-price, market share, rate of uptake.
The rule is to use only the prime dependencies, otherwise the link between assumptions, model, and output becomes blurred and the whole process becomes black box' and people lose interest.
Black Scholes (aka Real Options) Nobody really understands Black Scholes as applied to forecasts, so don't even go there.
And, in a bit more detail, the basis is traded options; thus, the purchaser of a traded option is buying the upside, while not taking on the downside for the price paid.
In those circumstances, volatility is a good thing and increases in volatility increase the value of the option in the Black Scholes model.
In the real world, increased volatility around an investment is a bad thing, so the basis does not hold.
Decision treesThis is a logical step-by-step process that links options with judgmental probabilities.
It's good for thinking logic through but can give a very broad range of outcomes.
It is also possible for even simple decision trees to produce over 100 terminal nodes and, since the sum of all the probabilities for all the terminal nodes should add up to 100%, most have a less than 1% probability, which does not fill the decision maker with confidence about the project or help them make a decision.
Again, do not show too much irrelevant data.
Risk Adjusted Net Present Value (rNPV) Almost the complete opposite of the above-and about as useful as the above, unless you are simply trying to produce a rank order of projects-everything is boiled down to one number.
It tells you nothing about the potential of the project, or what could drive or threaten the value.
Life cycle development will almost always deliver a more favorable rNPV than a new chemical entity, leading to a potentially imbalanced portfolio.
In summary, forecasts can be designed to provide a risk profile around the most likely outcome.
To achieve this, the forecaster needs to focus on the main uncertainties and use the risk management tools available in a simple, consistent and understandable way.
For more on the future of forecasting, join the sector's other key players at Forecasting Europe on June 14-15 in Berlin.
For all the latest business analysis and insight for the pharma industry, sign up to eyeforpharma's newsletters.