Can I test the validity of a prior given data?

Question

Problem

I am writing an R function that performs a Bayesian analysis to estimate a posterior density given an informed prior and data. I would like the function to send a warning if the user needs to reconsider the prior.

In this question, I am interested in learning how to evaluate a prior. Previous questions have covered the mechanics of stating informed priors ( here and here.)

The following cases might require that the prior be re-evaluated:

• the data represents an extreme case that was not accounted for when stating the prior
• errors in data (e.g. if data is in units of g when the prior is in kg)
• the wrong prior was chosen from a set of available priors because of a bug in the code

In the first case, the priors are usuallystill diffuse enough that the data will generally overwhelm them unless the data values lie in an unsupported range (e.g. <0 for logN or Gamma). The other cases are bugs or errors.

Questions

2. Are there any issues concerning the validity of using data to evaluate a prior?
3. is any particular test best suited for this problem?

Examples

Here are two data sets that are poorly matched to a $logN(0,1)$ prior because they are from populations with either $N(0,5)$ (red) or $N(8,0.5)$ (blue).

The blue data could be a valid prior + data combination whereas the red data would require a prior distribution that is supported for negative values.

 set.seed(1)
 x<- seq(0.01,15,by=0.1)
 plot(x, dlnorm(x), type = 'l', xlim = c(-15,15),xlab='',ylab='')
 points(rnorm(50,0,5),jitter(rep(0,50),factor =0.2), cex = 0.3, col = 'red')
 points(rnorm(50,8,0.5),jitter(rep(0,50),factor =0.4), cex = 0.3, col = 'blue')

Answer 1

You need to be clear what you mean by "prior". For example, if you are interested in my prior belief about the life expectancy in the UK, that can't be wrong. It's my belief! It can be inconsistent with the observed data, but that's another matter completely.

Also context matters. For example, suppose we are interested in the population of something. My prior asserts that this quantity must be strictly non-negative. However the data has been observed with error and we have negative measurements. In this case, the prior isn't invalid, it's just the prior for the latent process.

To answer your questions,

1. Are there any issues concerning the validity of using data to evaluate a prior?

A purist would argue that your shouldn't use the data twice. However, the pragmatic person would just counter that you hadn't thought enough about the prior in the first place.

2 Is any particular test best suited for this problem?

This really depends on the model under consideration. I suppose at the most basic you could compare prior range with data range.

Answer 2

Here my two cents:

1. I think you should be concerned about prior over parameters associated to ratios.

2. You talk about informative prior, but I think you should warn users about what a reasonable non-informative prior is. I mean, sometimes a normal with zero mean and 100 variance is fairly uninformative and sometimes it is informative, depending of the scales used. For instance, if you are regressing wages on heights (centimeters) than the above prior is quite informative. However, if you regressing log wages on heights (meters), then the above prior is not that informative.

3. If you are using a prior which is a result from a previous analysis, i.e, the new prior is actually an old posteriori of a previous analysis, then things are differente. I'm assuming this is note the case.