Eliciting priors from experts

Question

How should I elicit prior distributions from experts when fitting a Bayesian model?

Answer 1

John Cook gives some interesting recommendations. Basically, get percentiles/quantiles (not means or obscure scale parameters!) from the experts, and fit them with the appropriate distribution.

http://www.johndcook.com/blog/2010/01/31/parameters-from-percentiles/

Answer 2

I am currently researching the trial roulette method for my masters thesis as an elicitation technique. This is a graphical method that allows an expert to represent her subjective probability distribution for an uncertain quantity.

Experts are given counters (or what one can think of as casino chips) representing equal densities whose total would sum up to 1 - for example 20 chips of probability = 0.05 each. They are then instructed to arrange them on a pre-printed grid, with bins representing result intervals. Each column would represent their belief of the probability of getting the corresponding bin result.

Example: A student is asked to predict the mark in a future exam. The figure below shows a completed grid for the elicitation of a subjective probability distribution. The horizontal axis of the grid shows the possible bins (or mark intervals) that the student was asked to consider. The numbers in top row record the number of chips per bin. The completed grid (using a total of 20 chips) shows that the student believes there is a 30% chance that the mark will be between 60 and 64.9.

Some reasons in favour of using this technique are:

1. Many questions about the shape of the expert's subjective probability distribution can be answered without the need to pose a long series of questions to the expert - the statistician can simply read off density above or below any given point, or that between any two points.

2. During the elicitation process, the experts can move around the chips if unsatisfied with the way they placed them initially - thus they can be sure of the final result to be submitted.

3. It forces the expert to be coherent in the set of probabilities that are provided. If all the chips are used, the probabilities must sum to one.

4. Graphical methods seem to provide more accurate results, especially for participants with modest levels of statistical sophistication.

Answer 3

Eliciting priors is a tricky business.

Statistical Methods for Eliciting Probability Distributions and Eliciting Probability Distributions are quite good practical guides for prior elicitation. The process in both papers is outlined as follows:

1. background and preparation;
2. identifying and recruiting the expert(s);
3. motivation and training the expert(s);
4. structuring and decomposition (typically deciding precisely what variables should be elicited, and how to elicit joint distributions in the multivariate case);
5. the elicitation itself.

Of course, they also review how the elicitation results in information that may be fit to or otherwise define distributions (for instance, in the Bayesian context, Beta distributions), but quite importantly, they also address common pitfalls in modeling expert knowledge (anchoring, narrow and small-tailed distributions, etc.).

Answer 4

I'd recommend letting the subject expert specify the mean or mode of the prior but I'd feel free to adjust what they give as a scale. Most people are not very good at quantifying variance.

And I would definitely not let the expert determine the distribution family, in particular the tail thickness. For example, suppose you need a symmetric distribution for a prior. No one can specify their subjective belief so finely as to distinguish a normal distribution from, say, a Student-t distribution with 5 degrees of freedom. But in some contexts the t(5) prior is much more robust than the normal prior. In short, I think the choice of tail thickness is a technical statistical matter, not a matter of quantifying expert opinion.

Answer 5

I've had training in and used the SHELF framework, which does do some of the good things other responses have already mentioned. Many good practices are discussed in the materials available from the SHELF webpage and also this book on Eliciting Experts' Probabilities. Why am I writing another answer, when there's already other goods ones mentioning many good points? The main reason is that what's not mentioned much, yet, is the multivariate nature of the problem, which I think you cannot ignore.

The issue is that "prior distributions from experts when fitting a Bayesian model" would usually require you to obtain prior distributions for multiple model parameters. It turns out that you should not elicit priors for e.g. each coefficient of a regression model independently. You may get a sensible marginal prior for each coefficient, but you still end up with a combined prior belief that makes no sense and that you would reject. The reason is that extreme values in the independent priors on multiple coefficients "can combine" so as to assign way too much prior density to predicted outcomes that are almost impossible. Prior predictive checks certainly seem to be a good tool for getting a feeling for whether there is a problem in this respect.

To resolve this potential problem, it seems like one ought to elicit how the prior beliefs about different model parameters are correlated. With two parameters this is rather doable (arxiv link), but with more parameters it becomes complex (e.g. keeping elicited correlation matrices valid is tricky) and definitely a bit time consuming.

In contrast ideas like automatically setting correlation to keep the prior predictive distribution within a range elicited from experts would be easier to implement, but also requires an idea about the distribution of covariates. More importantly, I worry that this might ignore that the experts' beliefs about some coefficients might be much more strongly correlated than for others. If the experts can't really say or don't have any strong beliefs, then this might be a promising idea, because experts probably can say something about the the predictive distribution.

Answer 6

This interesting question is the subject of some research in ACERA$^\dagger. $ The lead researcher is Andrew Speirs-Bridge, and his work is eminently google-able :)

---

$^\dagger$ New link: 2