Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

How should I elicit prior distributions from experts when fitting a Bayesian model?

share|improve this question
4  
Although I've accepted an answer, I would recommend that interested people should look at all the answers. –  csgillespie Jul 27 '10 at 15:22

5 Answers 5

up vote 13 down vote accepted

John Cook gives some interesting recommendations. Basically, get percentiles/quantiles (not means or obscure scale parameters!) from the experts, and fit them with the appropriate distribution.

http://www.johndcook.com/blog/2010/01/31/parameters-from-percentiles/

share|improve this answer

I am currently researching the trial roulette method for my masters thesis as an elicitation technique. This is a graphical method that allows an expert to represent her subjective probability distribution for an uncertain quantity.

Experts are given counters (or what one can think of as casino chips) representing equal densities whose total would sum up to 1 - for example 20 chips of probability = 0.05 each. They are then instructed to arrange them on a pre-printed grid, with bins representing result intervals. Each column would represent their belief of the probability of getting the corresponding bin result.

Example: A student is asked to predict the mark in a future exam. The figure below shows a completed grid for the elicitation of a subjective probability distribution. The horizontal axis of the grid shows the possible bins (or mark intervals) that the student was asked to consider. The numbers in top row record the number of chips per bin. The completed grid (using a total of 20 chips) shows that the student believes there is a 30% chance that the mark will be between 60 and 64.9.

Eliciting priors from experts - Trial Roulette Method

Some reasons in favour of using this technique are:

  1. Many questions about the shape of the expert's subjective probability distribution can be answered without the need to pose a long series of questions to the expert - the statistician can simply read off density above or below any given point, or that between any two points.

  2. During the elicitation process, the experts can move around the chips if unsatisfied with the way they placed them initially - thus they can be sure of the final result to be submitted.

  3. It forces the expert to be coherent in the set of probabilities that are provided. If all the chips are used, the probabilities must sum to one.

  4. Graphical methods seem to provide more accurate results, especially for participants with modest levels of statistical sophistication.

share|improve this answer
3  
Cool! Please post a link here to your thesis once it's complete and/or published! –  Harlan Jul 21 '10 at 12:20
    
Looks very interesting indeed! –  AndyF Aug 14 '10 at 11:29
    
@Harlan I couldn't track down her thesis, but the trial roulette method is described on p 18 of Eliciting Probability Distributions (a nice reference pointed out by @john-l-taylor) –  Abe Jun 28 '13 at 14:50

Eliciting priors is a tricky business.

Statistical Methods for Eliciting Probability Distributions and Eliciting Probability Distributions are quite good practical guides for prior elicitation. The process in both papers is outlined as follows:

  1. background and preparation;
  2. identifying and recruiting the expert(s);
  3. motivation and training the expert(s);
  4. structuring and decomposition (typically deciding precisely what variables should be elicited, and how to elicit joint distributions in the multivariate case);
  5. the elicitation itself.

Of course, they also review how the elicitation results in information that may be fit to or otherwise define distributions (for instance, in the Bayesian context, Beta distributions), but quite importantly, they also address common pitfalls in modeling expert knowledge (anchoring, narrow and small-tailed distributions, etc.).

share|improve this answer

I'd recommend letting the subject expert specify the mean or mode of the prior but I'd feel free to adjust what they give as a scale. Most people are not very good at quantifying variance.

And I would definitely not let the expert determine the distribution family, in particular the tail thickness. For example, suppose you need a symmetric distribution for a prior. No one can specify their subjective belief so finely as to distinguish a normal distribution from, say, a Student-t distribution with 5 degrees of freedom. But in some contexts the t(5) prior is much more robust than the normal prior. In short, I think the choice of tail thickness is a technical statistical matter, not a matter of quantifying expert opinion.

share|improve this answer
    
excellent point about the tails, which I think is key. Another contrasting example would be to consider weibull and gamma as alternatives to the log-normal. In practice, these often provide more realistic fits to right-skewed positive variables. –  Abe Jun 28 '13 at 14:41

This interesting question is the subject of some research in ACERA. The lead researcher is Andrew Speirs-Bridge, and his work is eminently google-able :)

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.