Approaching uncertainty

It seems obvious. If you don’t have the answer then you ask an expert. But what if the expert doesn’t know?

How many decisions do we make every day? 10? 20? Not even close. From the moment we open our eyes in the morning to closing them again at night, our day is full of decisions. 35,000 to be exact. And we rely on experts to help us make them, from GPs managing our health to the information at our fingertips via search engines. But sometimes the experts don’t have the answers. Without the right information, or if there’s uncertainties in the knowledge we do have, making even the smallest decisions can become a challenge. It’s then that we need an alternative solution.

decision making graphic

This is how Jeremy Oakley and Tony O’Hagan came to develop SHELF, the Sheffield Elicitation Framework. Jeremy and Tony, who are Professors of Maths and Statistics at the University of Sheffield, developed SHELF to combat the uncertainties that can arise when making a decision. “It’s a process for capturing uncertainty and presenting it in a form that’s useful for decision-makers,” explains Jeremy.  

SHELF starts by collecting all the relevant evidence about the topic in question. Experts on the subject are then gathered and asked to make judgements on the possible outcomes based on the information. After a carefully managed discussion a consensus is reached which acknowledges both the agreements and disagreements between the experts. The process is designed to make sure that all views are heard and considered, and to understand why experts disagree.

SHELF has several features that make it stand out from other tools. Perhaps most importantly it combines the judgements of several experts into a final output. This output expresses the uncertainty and the vast knowledge of the group. Other methods ask experts questions that they already have the answer to and then gives a low rank to experts whose judgements are wrong. SHELF gets the experts to discuss their differences, which transforms the elicitation process.  

experts at a table

The framework has a broad range of applications, “SHELF can be used anywhere people are relying on expert opinion to do things,” Jeremy says. This is how GlaxoSmithKline (GSK) came to use SHELF. As a world top 10 pharmaceutical company, GSK turned over £17 billion in pharmaceutical products in 2017 and are leaders in drug development. But drug development is a costly process.

The average cost of a failed clinical trial varies, but it’s estimated at between £600 million and £1 billion across the pharmaceutical industry. This begs the question: how sure are we really that a trial is going to succeed? The catch-22 here is that it’s not possible to know how well a drug really works until the trial is completed, but you need to do the trial to find out if the drug really works. At a London conference in 2012, former CEO of GSK, Andrew Witty, commented on the topic of drug pricing, “If you stop failing so often you massively reduce the cost of drug development.”

This is where SHELF and expert judgement comes in. At each stage in the drug development process there’s a decision to be made about whether or not to progress. Once identified as safe, with the potential to be effective, the drug is evaluated against existing medications. Despite companies being led to believe that clinical trials will have as high as a 90% success rate, trends in the pharmaceutical industry show the success rate of phase three to be around 60-70%. In other words, up to a third of drugs fail to make it past this stage despite being safe and potentially effective.

Petri dish

Following the SHELF process, the experts pool their knowledge and evidence, think through the uncertainties, and pass on their assessment of how likely a planned trial is to succeed. By using SHELF to determine the likelihood of the trial succeeding, companies can save a substantial amount of time and money. “As a result of SHELF, lots of changes can be made to trial design,” says Nicky Best, the Head of Advanced Biostatistics and Data Analytics Centre of Excellence at GSK. GSK have put SHELF at the heart of their planning for clinical trials, and have already used it in 50 trials.

SHELF is now one of the gold standard tools for expert elicitation but it’s not alchemy. “If the experts really don’t know then that’s what SHELF will show,” says Jeremy. And it’ll certainly help with at least a few of those 35,000 decisions.

Further information


Relevant publications

Assurance in clinical trial design

Uncertain judgements: eliciting experts probability

Assurance calculations for planning clinical trials with time-to-event outcomes


Funders, awards and key grants

2003-2006: “Elicitation of individuals’ knowledge in probabilistic form”, National Health Service’s Research Methodology Programme


Related studies

Masters courses in Statistics

PhD opportunities


Online research profile URLs

Sheffield University Jeremy Oakley

Sheffield University Anthony O’Hagan

Anthony O’Hagan


For more information please contact: 

Alicia Shephard
Research Marketing and Content Coordinator 
University of Sheffield
0114 222 1033
alicia.shephard@sheffield.ac.uk