1
$\begingroup$

I have a question about prior choice that has arisen from some analysis I have been doing. I don't think the particular details of the model are necessary for this question, but my Bayesian knowledge is relatively limited so apologies if I have omitted something crucial.

I have run a model twice, once with a relatively uninformative prior, and one with a more informative prior. Both times the priors are exponential.

The likelihood for the uninformed model is higher than for the more informed model.

Is it appropriate to infer from this that the informed prior is a poor choice?

EDIT

Here are the MCMC chains from the two different analyses

1) Prior with exp(1)

enter image description here

2) Prior with exp(10)

enter image description here

I do think something strange is going on in both of these, but I don't know why they aren't in the same likelihood space.

$\endgroup$
  • 3
    $\begingroup$ The likelihood is based solely on the data and therefore is not affected by the prior. Perhaps you mean the "posterior is more peaked" when the uninformative prior is used? If this is the case, then I think you should provide details of the model and priors as this is an unusual situation. $\endgroup$ – jaradniemi Feb 6 '18 at 16:30
  • $\begingroup$ A comparison of two priors can be done via a Bayes factor, not by comparing directly the likelihoods (at which values? pluggin estimates?) $\endgroup$ – Xi'an Feb 6 '18 at 16:32
  • 2
    $\begingroup$ That can come about if, for example, the peak of the informed prior is located some ways away from the peak of the likelihood function. The probability of the informed prior in the region around where the likelihood is peaked may actually be lower than the probability of a relatively uninformed prior in the same region, consequently lowering the peak relative to the uninformed prior. Heuristically, it's harder to learn "the truth" if you start out wrong and relatively certain about it than if you start out wrong and uncertain about it. $\endgroup$ – jbowman Feb 6 '18 at 17:20
  • $\begingroup$ What variable is on the vertical axis in your MCMC chains? $\endgroup$ – Ben Feb 9 '18 at 21:38
  • $\begingroup$ The likelihood. $\endgroup$ – SamPassmore Feb 10 '18 at 8:45
1
$\begingroup$

You can report the influence a prior has on a result, but there is no way to distinguish "good or bad." The purpose of the prior is to include real information that you have about the likely location of the parameter. As an example, imagine that you had performed prior research on the topic and so you used those parameter estimates as your prior. As a comparison, you used an uninformative prior to show the difference.

Now let's consider a couple of possible scenarios. In the first scenario, unknown to you, in your first sample for your first set of experiments, you, by chance, chose a very unusual sample if you had known the true sample space. The informed prior knowledge happens to be bad. The second sample is reasonably representative. You don't know what to do, so you drop the informative prior from your research. As it works out, this is fortunate because the happenstance of drawing an unusual sample is now avoided.

In the second example, unknown to you, the first sample happens to be very representative of the data and you use the resulting posterior as your prior. Unfortunately, when you perform the second experiment, by chance, you happen to draw a very unusual sample. You compare the results with the two priors and drop the one with the informative prior. Unfortunately, you now have a very biased set of parameter estimates because you are unaware that you have drawn a strange sample.

If the informed prior really represents information from the literature or experience about the actual location of the prior, then it should be left alone. It may be the case that the content in the original experiment was a false discovery and so you may modify or weaken your prior to allow for that, but it should be maintained in the research.

You should have constructed your prior density before doing the experiment as it contains the information about the parameters from outside the sample itself.

The question you should be asking yourself is why and how you formed your more informative prior. If you believe that information is still good, then disclose your method and reasoning for its construction and contrast it with what you observed in the less informative prior density.

$\endgroup$
  • 1
    $\begingroup$ Would looking at how well you can a-posteriori predict new data you have not seen, yet, be a reasonable way to distinguish how well a prior performs? $\endgroup$ – Björn Feb 14 '18 at 8:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.