Elicitation of Experts' Probabilities

The Problem

Elicitation is the process of extracting expert knowledge about some unknown quantity of interest, or the probability of some future event, which can then be used to supplement any numerical data that we may have. If the expert in question does not have a statistical background, as is often the case, translating their beliefs into a statistical form suitable for use in our analyses can be a challenging task.

Prior elicitation is an important and yet under researched component of Bayesian statistics. In any statistical analysis there will typically be some form of background knowledge available in addition to the data at hand. For example, suppose we are investigating the average lifetime of a component in some piece of machinery. We can do tests on a sample of components to learn about their average lifetime, but the designer of the component may have their own expectations about its performance. If we can represent the expert's uncertainty about the lifetime through a probability distribution, then this additional (prior) knowledge can be utilized within the Bayesian framework.

In practice, prior knowledge is often not considered in Bayesian analyses. With a large quantity of data, prior knowledge tends to have less of an effect on final inferences. Given this fact, and the various techniques available for representing prior ignorance, practitioners of Bayesian statistics are frequently spared the effort of thinking about the available prior knowledge.

It will not always be the case that we will have sufficient data to be able to ignore prior knowledge, and one example of this would be in the uncertainty in computer models application, described in the Bayesian cluster pages. Uncertain model input parameters are often assigned probability distributions entirely on the basis of expert judgments. In addition, certain parameters in statistical models can be hard to make inferences about, even with a reasonable amount of data.

The volume of research in eliciting prior knowledge is relatively low, and various proposed techniques are often targeted at specific applications. At the same time, recent advances in Bayesian computation have allowed far greater flexibility in modelling prior knowledge. In general, elicitation can be made difficult by the fact that we cannot expect the expert to provide probability distributions for quantities of interest directly. The challenge is then to find appropriate questions to ask the expert in order to extract their knowledge, and then to determine a suitable probabilistic description of the variables we are interested in based on the information we have learned from them.

Research at CHEBS

Research in this area is being carried out by Tony O'Hagan, Jeremy Oakley and Caitlin Buck. It has received funding recently from the Department of Health for the project Bayesian Elicitation of Experts´ Probabilities. This has led to the publication in 2006 of a book, Uncertain Judgements, reviewing the literature and identifying best current practice and topics for research.