10
$\begingroup$

Problem

I am writing an R function that performs a Bayesian analysis to estimate a posterior density given an informed prior and data. I would like the function to send a warning if the user needs to reconsider the prior.

In this question, I am interested in learning how to evaluate a prior. Previous questions have covered the mechanics of stating informed priors ( here and here.)

The following cases might require that the prior be re-evaluated:

  • the data represents an extreme case that was not accounted for when stating the prior
  • errors in data (e.g. if data is in units of g when the prior is in kg)
  • the wrong prior was chosen from a set of available priors because of a bug in the code

In the first case, the priors are usuallystill diffuse enough that the data will generally overwhelm them unless the data values lie in an unsupported range (e.g. <0 for logN or Gamma). The other cases are bugs or errors.

Questions

  1. Are there any issues concerning the validity of using data to evaluate a prior?
  2. is any particular test best suited for this problem?

Examples

Here are two data sets that are poorly matched to a $logN(0,1)$ prior because they are from populations with either $N(0,5)$ (red) or $N(8,0.5)$ (blue).

The blue data could be a valid prior + data combination whereas the red data would require a prior distribution that is supported for negative values.

enter image description here

 set.seed(1)
 x<- seq(0.01,15,by=0.1)
 plot(x, dlnorm(x), type = 'l', xlim = c(-15,15),xlab='',ylab='')
 points(rnorm(50,0,5),jitter(rep(0,50),factor =0.2), cex = 0.3, col = 'red')
 points(rnorm(50,8,0.5),jitter(rep(0,50),factor =0.4), cex = 0.3, col = 'blue')
$\endgroup$
4
$\begingroup$

You need to be clear what you mean by "prior". For example, if you are interested in my prior belief about the life expectancy in the UK, that can't be wrong. It's my belief! It can be inconsistent with the observed data, but that's another matter completely.

Also context matters. For example, suppose we are interested in the population of something. My prior asserts that this quantity must be strictly non-negative. However the data has been observed with error and we have negative measurements. In this case, the prior isn't invalid, it's just the prior for the latent process.

To answer your questions,

  1. Are there any issues concerning the validity of using data to evaluate a prior?

A purist would argue that your shouldn't use the data twice. However, the pragmatic person would just counter that you hadn't thought enough about the prior in the first place.

2 Is any particular test best suited for this problem?

This really depends on the model under consideration. I suppose at the most basic you could compare prior range with data range.

$\endgroup$
  • $\begingroup$ thanks for your answer, especially to #1 is helpful. For the test, I had thought of that, but the range of most priors will have a bound at $\infty$, so I was thinking of perhaps comparing the bounds of quantile intervals, e.g. send warning if: the 80th quantile of data > 99th quantile of the prior or if: any data is greater than the 100-10e-log(n)th quantile) although I would have to play around with the numbers so that I catch the right errors. $\endgroup$ – David LeBauer Jun 23 '11 at 20:35
3
$\begingroup$

Here my two cents:

  1. I think you should be concerned about prior over parameters associated to ratios.

  2. You talk about informative prior, but I think you should warn users about what a reasonable non-informative prior is. I mean, sometimes a normal with zero mean and 100 variance is fairly uninformative and sometimes it is informative, depending of the scales used. For instance, if you are regressing wages on heights (centimeters) than the above prior is quite informative. However, if you regressing log wages on heights (meters), then the above prior is not that informative.

  3. If you are using a prior which is a result from a previous analysis, i.e, the new prior is actually an old posteriori of a previous analysis, then things are differente. I'm assuming this is note the case.

$\endgroup$
  • $\begingroup$ could you please clarify point 1? re: point 2, As mentioned in the OP, I am not so interested in this question about how to set the prior; re point 3: many of the informed priors are from analysis of available data (fitting a suitable distribution to data) whereas others are based on expert knowledge (these are generally less constrained). $\endgroup$ – David LeBauer Jun 23 '11 at 21:38
  • $\begingroup$ Assume you are fitting a model like: y ~ a + b*x/z. If there is no constraint on the values of Z (if they can be positive or negative), than it's hard to know what expect about the signal from b. Moreover, if Z can be near zero, than b can be too low or too big. This can make your prior unreasonable. See this entry on Gelman's blog: stat.columbia.edu/~cook/movabletype/archives/2011/06/… $\endgroup$ – Manoel Galdino Jun 24 '11 at 1:45
  • $\begingroup$ #3: As pointed, be careful about using the data twice. On thin is a hierarchical model, for instance, and another one is to pick a prior that is in agreement with the likelihood. In the later, I'd be concerned with such analysis. I see the choice of a prior more as a regularization tool. $\endgroup$ – Manoel Galdino Jun 24 '11 at 1:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.