# Prior distribution importance in Bayesian inference

I am performing a Bayesian multivariate regression, and therefore I have to construct the prior and the subsequent posterior.

But the paper that I am using as a reference, uses a "Uninformative prior", and moreover, gives me directly the expression of the posterior.

So what I did, was just drawing from the posterior directly, without considering the prior at all.

A detail of what I meant is this:

A standard uninformative prior here is

$$p(C, \Sigma) \propto |\Sigma|^{-(n+2)/2}$$

the posterior $p(C, \Sigma^{-1}\mid z)$ is given by

$$\Sigma^{-1}\mid z \sim \mathrm{Wishart}(T-n-2, S^{-1}) \\ vec(C)\mid \Sigma,z \sim N(vec(\hat C), (X'X)^{-1})$$

I just draw from the posterior (so the last two expressions).

I didn't understood the meaning of uninformative prior, and if, since it gives me no information, I can "skip it" and use directly the posterior derived from it.

Let me know if my reasoning is correct, or if you can suggest me how to incorporate the prior (how is that distribution defined)?

$$\text{posterior} \propto \text{prior} \times \text{likelihood}$$

so posterior combines information that comes from your data (through likelihood) and information that comes from your prior. So there is no posterior without prior. In case of the quote you provided, it provides a closed-form posterior distribution, because it uses the fact that Wishart distribution is a conjugate prior for precision parameter of multivariate normal distribution, so we have a "standalone" closed-form for it.

This means that for such model you do not have to sample from the prior distribution to conduct a MCMC simulation, since the posterior is known.

This does not mean that we do not use prior in here, we have already used it. On another hand, if you do not want to use prior, then you should rather use maximum likelihood estimation rather then Bayesian estimation.

• Exactly, this is exactly what I did. When i mean "not used"i was meaning that it was already incorporated in the posterior, and as a such there was no need to explicitly draw from it, but it was enough to consider directly the posterior to start the MCMC simulation. – Dave92 Dec 18 '16 at 20:52
• You don't need to sample, the probability surface is known exactly. – Dave Harris Dec 18 '16 at 23:29

The posterior already incorporates the prior through Bayes rule; that's the whole point of it being a posterior.

"Uninformative" has many different mathematical definitions (see their reference 12 for whatever definition of uninformative prior was used here), but all definitions refer to some sort of broad, vague prior that has minimal influence on the posterior.

• Ok, apparently I am receiving contrasting answers. What you said was exactly how I interpreted it, so I was drawing directly from the posterior. But since I am getting wrong results maybe is because of that – Dave92 Dec 18 '16 at 18:43

No, you cannot skip it. Uninformative priors contain information and for multivariate regression assure that the sum of the probabilities will be unity. In fact, you cannot use a uniform prior on a multivariate regression with three or more independent variables or the sum of the probabilities of your posterior will not equal one. This is likely the reason that Stein's lemma in Frequentist statistics exists. If you try it and you get lucky, then you will get screwy results that warn you something is wrong. If you get unlucky, the results will not permit you to detect the problems.

Let me give you some examples of the information in uninformative prior distributions. For the binomial distribution whose true parameter value is unknown then a simple prior is the beta distribution. This does not mean you should use a beta distribution, merely that it is very convenient. There are three known uninformative priors for the binomial, each is a form of the beta distribution. The first will result in an point estimate identical to the Frequentist solution. It is $$p^{-1}(1-p)^{-1}.$$ It provides an unbiased estimator for $p$ in the sense that the maximum a posteriori(MAP) estimator and the Frequentist estimator match for the estimated value of $p$.

If you look at that prior, however, it provides infinite weight on either zero or one and minimal weight to $p=\frac{1}{2}$. It is uninformative in the sense that it does not influence the location of the MAP estimator. It does not weight the tails uniformly though.

The second is the Jeffreys' prior, which is $$p^{-1/2}(1-p)^{-1/2}.$$ As with the first one, it provides infinite weight on either extreme and minimal weight at $p=1/2$. It is a biased estimator in that it adds one half a success and one half of a failure to the ultimate solution. It is equivalent to tossing a coin one time and it lands on its side. The reason to use it is that it permits you to do something that cannot be done in non-Bayesian methods and cannot be done with either uninformative prior, it allows you to transform the variable of interest and get the same statistical results under the transformation as you get in the raw data.

A common transformation is to take the logarithm of data, but $\hat{\mu}_{raw}\ne\exp(\log(\hat{\mu}_{\log}))$ in most cases. If you are careful in your use of the prior, then $\hat{\mu}_{raw}=\exp(\log(\hat{\mu}_{\log}))$, as well as all other moments and intervals. A Jeffreys' prior preserves results across transformations. An important area of research in Bayesian methods are how to create invariant results.

The third common uninformative prior is the uniform distribution, which is also a beta distribution. The uniform distribution is $\Pr(p)=1,\forall{p}\in(0,1)$. This is equivalent to adding one success and one failure to the final answer. A consequence of this is that although each possible solution is equally probable, prior to seeing the data, the expectation of $p$ will be biased toward the center.

This is also true for multivariate regression. The more important issue, however, is whether or not you are in possession of prior information. Is there no prior research among any of the variables? It is generally the case that prior information exists, the more important issue is the disciplined incorporation of prior learning.

Let me give you a silly example, that at another level, is not so silly. Imagine you were eating green beans and you wondered how many calories were in the forkful you were holding. You decide to be careful and sample from the can of beans in equal amounts. You have many samples. Strangely, you also own a calorimeter. You want to estimate the calories per gram.

At first you consider using the maximum likelihood estimate(MLE), but then you realize that you could do better. The MLE is equivalent to using a uniform prior over the half plane, since you also have to estimate standard deviation as a nuisance parameter. You realize that you cannot have negative calories, so you only need a quarter plane. That is information, even though it gives equal weight to all positive solutions.

Then you realize you could do better. The recommended intake for an adult male is 2000 calories and you do not believe that a person could survive on one forkful of green beans, so you restrict it to uniform up to 2000 calories and half normal above 2000 calories so it quickly tapers off. Of course, you then realize the can itself tells the FDA estimate of the calories, but you are aware there have been significant errors in the FDA estimate in specific isolated foods.

According to the FDA there are .31 calories per gram, but how big should the standard deviation be? You don't trust the FDA estimate, so you reason that if three standard deviations are to the left, you should allow three more to the right so you have a truncated normal whose $\pm{3}\sigma=.31$ and whose center equals .31.

You have just gone from an uninformative prior to a weakly informative prior. If you collected multiple studies, you could weight them into your prior for a strongly informative prior distribution. This would protect you from happening to get the weird sample of beans by random chance.

The prior is the most difficult thing to define, particularly if you have academic adversaries as it should be no stronger than their beliefs if you are to convince them. It sounds like you are using conjugate priors in order to get mathematical simplicity.

Although there are giant discussions on this, I will give you a weak heuristic to find your prior. First, gather as much academic research, Frequentist or Bayesian, on the topic that you can gather. You are looking for sample size, slope, intercept, covariance and variance estimates. If you think a study was very poorly done, then multiply its variance/covariance estimate by some suitable number, such as 25 or 100. This will weaken its influence in the process, but preserve the estimate of the location. You are keeping poor data, by weakening its value, but not excluding it entirely. Consider that your first prior, then look at the second study. That study, assuming they do not use the same data set, will be your likelihood. If you are using conjugate priors, update this so that your posterior is now the product of the two studies. Now get your third study and treat your last posterior as your new prior and the third study as your likelihood to get a new posterior. Continue this until you run out of studies and the final posterior will be your prior.

Again, if the research isn't exactly your questions then multiply the variances and covariances by some large enough number that it weakens the impact of the prior research. If some of the research uses different variables, then you will have to make judgment calls as there is no simple solution.

If you are wed to an uninformative prior because you are afraid of criticism, then you want the parameters of the prior to make a distribution that is as spread out as possible, but still a proper distribution. That is to say, it adds to one. You can, for example, give a million unit standard deviation when you believe it will probably be .01 units. If the prior is diffuse enough, it won't matter, except that it trivially avoids the problem brought about in Stein's lemma.

A solution is to look at the smallest reported digit. If your report out to five digits past the decimal, then you want your variance to be large enough that it only impacts up to the sixth decimal in the calculation.

There are far better solutions than this, but you really need to sit down with a statistician to do them. Bayesian methods are not DIY methods at first.

• Thanks for the answer! But given that accounting for the prior is needed, do you have any idea of how to define it? Honestly I have no idea of how to define it. In the sense that abs(Sigma)^(-n+2)/n) does not look familiar to me honestly – Dave92 Dec 18 '16 at 18:41
• You are solving for the slopes and the precisions, just be clear using the above model and not the slope and the variances/covariances. This is okay, but you need to realize the distinction as $\rho=1/\sigma^2.$ You should pick up a basic textbook such as Bolstad's followed by a slightly more advanced one such as the one by Peter Lee. – Dave Harris Dec 18 '16 at 18:54
• people.eecs.berkeley.edu/~jordan/courses/260-spring10/lectures/… solves the problem using variances and goes further out than your specific request. – Dave Harris Dec 18 '16 at 18:55
• thaines.com/content/misc/… is the problem that you are solving. Remember, when you are using the Wishart you are solving for precision and not variance. Precisions are additive, which is why they are used. – Dave Harris Dec 18 '16 at 18:55
• Just a note, that is not an absolute value, it is a determinant. – Dave Harris Dec 18 '16 at 18:59