Eliciting priors from experts

Question

How should I elicit prior distributions from experts when fitting a Bayesian model?

Answer 1

John Cook gives some interesting recommendations. Basically, get percentiles/quantiles (not means or obscure scale parameters!) from the experts, and fit them with the appropriate distribution.

http://www.johndcook.com/blog/2010/01/31/parameters-from-percentiles/

Answer 2

I am currently researching the trial roulette method for my masters thesis as an elicitation technique. This is a graphical method that allows an expert to represent her subjective probability distribution for an uncertain quantity.

Experts are given counters (or what one can think of as casino chips) representing equal densities whose total would sum up to 1 - for example 20 chips of probability = 0.05 each. They are then instructed to arrange them on a pre-printed grid, with bins representing result intervals. Each column would represent their belief of the probability of getting the corresponding bin result.

Example: A student is asked to predict the mark in a future exam. The figure below shows a completed grid for the elicitation of a subjective probability distribution. The horizontal axis of the grid shows the possible bins (or mark intervals) that the student was asked to consider. The numbers in top row record the number of chips per bin. The completed grid (using a total of 20 chips) shows that the student believes there is a 30% chance that the mark will be between 60 and 64.9.

Some reasons in favour of using this technique are:

1. Many questions about the shape of the expert's subjective probability distribution can be answered without the need to pose a long series of questions to the expert - the statistician can simply read off density above or below any given point, or that between any two points.

2. During the elicitation process, the experts can move around the chips if unsatisfied with the way they placed them initially - thus they can be sure of the final result to be submitted.

3. It forces the expert to be coherent in the set of probabilities that are provided. If all the chips are used, the probabilities must sum to one.

4. Graphical methods seem to provide more accurate results, especially for participants with modest levels of statistical sophistication.

Answer 3

Eliciting priors is a tricky business.

Statistical Methods for Eliciting Probability Distributions and Eliciting Probability Distributions are quite good practical guides for prior elicitation. The process in both papers is outlined as follows:

1. background and preparation;
2. identifying and recruiting the expert(s);
3. motivation and training the expert(s);
4. structuring and decomposition (typically deciding precisely what variables should be elicited, and how to elicit joint distributions in the multivariate case);
5. the elicitation itself.

Of course, they also review how the elicitation results in information that may be fit to or otherwise define distributions (for instance, in the Bayesian context, Beta distributions), but quite importantly, they also address common pitfalls in modeling expert knowledge (anchoring, narrow and small-tailed distributions, etc.).

Answer 4

I'd recommend letting the subject expert specify the mean or mode of the prior but I'd feel free to adjust what they give as a scale. Most people are not very good at quantifying variance.

And I would definitely not let the expert determine the distribution family, in particular the tail thickness. For example, suppose you need a symmetric distribution for a prior. No one can specify their subjective belief so finely as to distinguish a normal distribution from, say, a Student-t distribution with 5 degrees of freedom. But in some contexts the t(5) prior is much more robust than the normal prior. In short, I think the choice of tail thickness is a technical statistical matter, not a matter of quantifying expert opinion.

Answer 5

This interesting question is the subject of some research in ACERA. The lead researcher is Andrew Speirs-Bridge, and his work is eminently google-able :)