Is it ok to fit a Bayesian model first, then begin weakening priors?

Question

When doing frequentist stats, there is a long list of big no-nos, like looking at the results of statistical tests before deciding to collect more data. I'm wondering generally if there are a similar list of no-nos for the methodologies involved in Bayesian statistics, and specifically whether the following is one of them.

I've realized recently that for some of the models I've been fitting, my process has been to first fit the model with informative priors to see if it works or blows up, and then weaken the priors either to uninformative or weakly informative and refit the model.

My motivation for this really has to do with the fact that I'm writing these models in JAGS/Stan, and in my mind I've been treating it more like a programming task than a statistical one. So, I do a first run, sort of rigging it to converge quickly by using informative priors, making it easier to catch errors in the model I've written. Then, after debugging the model, I refit it with uninformative, or weakly informative priors.

My question is whether or not I'm breaking some serious rules with this process. For example, for my inferences to be valid, and to avoid exploiting researchers' degrees of freedom, do I need to commit to specific priors before are start fitting any models?

Answer 1

Subjective Bayesians might disagree, but from my perspective, the prior is just a part of the model, like the likelihood. Changing the prior in response to model behavior is no better or worse than changing your likelihood function (e.g. trying different error distributions or different model formulations).

It can be dangerous if it lets you go on a fishing expedition, but the alternatives can be worse. For example, in the case you mentioned, where your model blows up and you get nonsensical coefficients, then you don't have much choice but to try again.

Also, there are steps you can take to minimize the dangers of a fishing expedition somewhat:

• Deciding in advance which prior you'll use in the final analysis
• Being up-front when you publish or describe your analysis about your whole procedure
• Doing as much as possible with either simulated data and/or holding out data for the final analysis. That way, you won't contaminate your analysis too much.

Answer 2

I think you're okay in this case for three reasons:

1. You're not actually adjusting your priors in response to your results. If you said something like, "I use XYZ priors and depending on the rate of convergence and my DIC results, I then modify my prior by ABC," then I'd say you were committing a no-no, but in this case it sounds like you really are not doing that.

2. In a Bayesian context, priors are explicit. So it's possible for you to tweak your priors improperly, but the resultant priors will always be visible for inspection by others who can question why you have those particular priors. Maybe I'm naive here, since it's easy to glance at something like a prior and say, "Hmm, looks reasonable" simply because someone offered it up, but...

3. I think what you're doing is related to Gelman's (and others') advice to build up a JAGS model piece-by-piece, first working with synthetic data, then real data, to make sure you don't have a specification error. That's not really a factor in frequentist methodology, and it's not really an experimental methodology.

Then again, I'm still learning this stuff myself.

P.S. When you say you originally rig it to converge quickly with "informative priors", do you mean actually informative priors that are motivated by the problem at hand, or just priors that for arbitrary reasons strongly push/restrict the posterior to speed up "convergence" to some arbitrary point? If it's the first case, why are you then moving away from these (motivated) priors?

Answer 3

If you experiment with priors and select one in terms of its performances on the data at hand, it is no longer a "prior". Not only does it depend on the data (as in an empirical Bayes analysis), but it also depends on what you want to see (which is worse). In the end, you do use Bayesian tools, but this cannot be called a Bayesian analysis.

Answer 4

I would say no, you do not have to commit to specific priors. Generally during any Bayesian data analysis you should perform an analysis of the sensitivity of the model to the prior. That would include trying various other priors to see what happens to the results. This might reveal a better or more robust prior that should be used.

The two obvious "no-no's" are: playing around with the prior too much to get a better fit, resulting in over fit and changing the other parameters of the model to get a better fit. As an example of the first: changing an initial prior on the mean so that it is closer to the sample mean. For the second: changing your explanatory variables/ features in a regression to get a better fit. This is a problem in any version of regression and basically invalidates your degrees of freedom.

Answer 5

I think this might be a no no independent of the Bayesian school. Jeffreys would want to use noninformative priors. Lindley might want you to use informative priors. Empirical Bayesians would ask that you let the data influence the prior. But I think although each school is making a different suggestion about the choice of prior, they all have an approach that does not mean that you can take the prior and keep tweaking it until you get the results you want. That would definitely be like looking at the data and contnuing to collect data and test until you reach your preconceived notion of what the answer should be.

Frequentist or Bayesian it doesn't matter i don't think anyone would want you to play tricks with (or massage) the data. Maybe this is something we all can agree on and Peter's funny poem is really apropo.