# Weakly informative prior distributions for scale parameters

I have been using log normal distributions as prior distributions for scale parameters (for normal distributions, t distributions etc.) when I have a rough idea about what the scale should be, but want to err on the side of saying I don't know much about it. I use it because the that use makes intuitive sense to me, but I haven't seen others use it. Are there any hidden dangers to this?

• Normals have conjugate priors: en.wikipedia.org/wiki/Normal-gamma_distribution . You might find these much easier to use. – whuber Jan 24 '11 at 19:26
• Interesting. I am doing numerical stuff, is there an advantage to these distributions besides congugality? – John Salvatier Jan 24 '11 at 19:40
• Not really my area but this 'might' be relevant? Gelman A. Prior distributions for variance parameters in hierarchical models. Bayesian Analysis 2006; 1:515–533. dx.doi.org/10.1214/06-BA117A – onestop Jan 24 '11 at 20:22
• I have found this Scaled-Beta$_2$ distribution proposed by Pérez and Pericchi. – user10525 May 10 '12 at 11:16
• Conjugate priors for a particular distribution like the normal are just priors that lead to that distribution as a posterior distribution given a set of data. If you use a conjugate prior you don't have to get into the mess of doing the integration to calculate the posterior. It makes things convenient but these days MCMC makes it much easier to use a wide variety of possible priors. – Michael R. Chernick May 10 '12 at 20:20

I would recommend using a "Beta distribution of the second kind" (Beta2 for short) for a mildly informative distribution, and to use the conjugate inverse gamma distribution if you have strong prior beliefs. The reason I say this is that the conjugate prior is non-robust in the sense that, if the prior and data conflict, the prior has an unbounded influence on the posterior distribution. Such behaviour is what I would call "dogmatic", and not justified by mild prior information.

The property which determines robustness is the tail-behaviour of the prior and of the likelihood. A very good article outlining the technical details is here. For example, a likelihood can be chosen (say a t-distribution) such that as an observation $y_i \rightarrow \infty$ (i.e. becomes arbitrarily large) it is discarded from the analysis of a location parameter (much in the same way that you would intuitively do with such an observation). The rate of "discarding" depends on how heavy the tails of the distribution are.

Some slides which show an application in the hierarchical modelling context can be found here (shows the mathematical form of the Beta2 distribution), with a paper here.

If you are not in the hierarchical modeling context, then I would suggest comparing the posterior (or whatever results you are creating) but use the Jeffreys prior for a scale parameter, which is given by $p(\sigma)\propto\frac{1}{\sigma}$. This can be created as a limit of the Beta2 density as both its parameters converge to zero. For an approximation you could use small values. But I would try to work out the solution analytically if at all possible (and if not a complete analytical solution, get the analytical solution as far progressed as you possibly can), because you will not only save yourself some computational time, but you are also likely to understand what is happening in your model better.

A further alternative is to specify your prior information in the form of constraints (mean equal to $M$, variance equal to $V$, IQR equal to $IQR$, etc. with the values of $M,V,IQR$ specified by yourself), and then use the maximum entropy distribution (search any work by Edwin Jaynes or Larry Bretthorst for a good explanation of what Maximum Entropy is and what it is not) with respect to Jeffreys' "invariant measure" $m(\sigma)=\frac{1}{\sigma}$.

MaxEnt is the "Rolls Royce" version, while the Beta2 is more a "sedan" version. The reason for this is that the MaxEnt distribution "assumes the least" subject to the constraints you have put into it (e.g., no constraints means you just get the Jeffreys prior), whereas the Beta2 distribution may contain some "hidden" features which may or may not be desirable in your specific case (e.g., if the prior information is more reliable than the data, then Beta2 is bad).

The other nice property of MaxEnt distribution is that if there are no unspecified constraints operating in the data generating mechanism then the MaxEnt distribution is overwhelmingly the most likely distribution that you will see (we're talking odds way over billions and trillions to one). Therefore, if the distribution you see is not the MaxEnt one, then there is likely additional constraints which you have not specified operating on the true process, and the observed values can provide a clue as to what that constraint might be.

• @probabilityislogic Nice answer. Do you know where can I find the papers you mention int the the third paragraph? The links are not working. – user10525 May 10 '12 at 9:04
• one that works for the paper is here. It was on a conference "objective bayes 09" website (the Valencia meetings). I don't think the slides will be available anymore, as the website for the conference has been taken down... :( pity, it was a good set of slides. That horshoe prior does look interesting in the link you provided. – probabilityislogic May 10 '12 at 21:29
• @probabilityislogic Perhaps I am missing something but I cannot find a reference to the $Beta_2$ in the BA paper. – user10525 May 11 '12 at 10:16
• @Procrastinator Am I right to assume that you want only proper priors ? You didn't say it but if you allow improper priors the already mentioned Jeffreys' priors would work and I could cite Jeffreys theory of probability, books by Dennis Lindley or statistics encyclopedia's. The way the request one could check using Google to find the answer and if it can't be found there is probably nothing in the literature outside the ones you have excleded. – Michael R. Chernick May 12 '12 at 19:47
• @MichaelChernick Yes, you are right, I am interested only in proper priors. The reason for this is that for proper priors (1) the existence of the posterior is not restrictive to certain models and (2) I wanted to check if I am not missing another interesting proposal. I agree with you that it seems like Gelman's, Pericchi's and Gamma priors are the most popular in the literature but I have also noted that there is a trend in proposing heavy-tailed priors in order to produce 'robust' inferences. – user10525 May 12 '12 at 20:32

The following paper by Daniels compares a variety of shrinkage priors for the variance. These are proper priors but I am not sure how many could be called non-informative if any. But, he also provides a list of noninformative priors (not all proper). Below is the reference.

M. J. Daniels (1999), A prior for the variance in hierarchical models, Canadian J. Stat., vol. 27, no. 3, pp. 567–578.

Priors

1. Flat: $K$ (constant)
2. Location-scale: $\tau^{-2}$
3. Right-invariant Haar: $\tau^{-1}$
4. Jeffreys': $1/(\sigma^2 + \tau^2)$
5. Proper Jeffreys': $\sigma / (2(\sigma^2 + \tau^2)^{3/2})$
6. Uniform shrinkage: $\sigma^2 / (\sigma^2 + \tau^2)$
7. DuMouchel: $\sigma/(2\tau(\sigma+\tau)^2)$

Another more recent paper in a related vein is the following.

A. Gelman (2006), Prior distributions for variance parameters in hierarchical models, Bayesian Analysis, vol. 1, no. 3, pp. 515–533.

• (+1) This is a good find. I've added a stable link to the Daniels paper as well as another reference that seems to complement it. – cardinal May 13 '12 at 23:46

(The question is stale, but the issue is not)

Personally, I think your intuition makes some sense. That is to say, if you don't need the mathematical tidiness of conjugacy, then whatever distribution you would use for a location parameter, you should use the same one for the log of a scale parameter. So, what you're saying is: use the equivalent of a normal prior.

Would you actually use a normal prior for a location parameter? Most people would say that, unless you make the variance huge, that's probably a bit "too dogmatic", for reasons explained in the other answers here (unbounded influence). An exception would be if you're doing empirical bayes; that is, using your data to estimate the parameters of your prior.

If you want to be "weakly informative", you'd probably choose a distribution with fatter tails; the obvious candidates are t distributions. Gelman's latest advice seems to be to use a t with df of 3-7. (Note that the link also supports my suggestion that you want to do the same thing for log of scale that you would do for location) So instead of a lognormal, you could use a log-student-t. To accomplish this in stan, you might do something like:

real log_sigma_y; //declare at the top of your model block
//...some more code for your model
log_sigma_y <- log(sigma_y); increment_log_prob(-log_sigma_y);
log_sigma_y ~ student_t(3,1,3); //This is a 'weakly informative prior'.


However, I think that if the code above is too complex for you, you could probably get away with a lognormal prior, with two caveats. First, make the variance of that prior a few times wider than your rough guess of how "unsure you are"; you want a weakly informative prior, not a strongly informative one. And second, once you fit your model, check the posterior median of the parameter, and make sure the log of it is not too far from the center of the lognormal. "Not too far" probably means: less than two standard deviations, and preferably not much more than one SD.

For hierarchical model scale parameters, I have mostly ended up using Andrew Gelman's suggestion of using a folded, noncentral t-distribution. This has worked pretty decently for me.