10
$\begingroup$

My actual questions are in the last two paragraphs, but to motivate them:

If I am attempting to estimate the mean of a random variable that follows a Normal distribution with a known variance, I've read that putting a uniform prior on the mean results in a posterior distribution that is proportional to the likelihood function. In these situations, the Bayesian credible interval overlaps perfectly with the frequentist confidence interval, and the Bayesian maximum a posteriori estimate is equal to the frequentist maximum likelihood estimate.

In a simple linear regression setting,

$Y = \textbf{X}\beta+\epsilon, \hspace{1cm} \epsilon\sim N(0,\sigma^2) $

putting a uniform prior on $\beta$ and an inverse-gamma prior on $\sigma^2$ with small parameter values results in a posterior $\hat\beta^{MAP}$ that will be very similar to the frequentist $\hat\beta^{MLE}$, and a credible interval for the posterior distribution of $\beta|X$ that will be very similar to the confidence interval around the maximum likelihood estimate. They will not be exactly the same because the prior on $\sigma^2$ exerts a small amount of influence, and if the posterior estimation is carried out via MCMC simulation that will introduce another source of discrepancy, but the Bayesian credible interval around the $\hat\beta^{MAP}$ and frequentist confidence interval around $\hat\beta^{MLE}$ will be pretty close to each other, and of course as the sample size increases they should converge as the influence of the likelihood grows to dominate that of the prior.

But I've read that there are also regression situations where these near-equivalencies do not hold. For example, hierarchical regressions with random effects, or logistic regression -- these are situations where, as I understand it, there are no "good" objective or reference priors.

So my general question is this -- assuming that I want to make inference about $P(\beta|X)$ and that I don't have prior information that I want to incorporate, why can't I proceed with frequentist maximum likelihood estimation in these situations and interpret the resulting coefficient estimates and standard errors as Bayesian MAP estimates and standard deviations, and implicitly treat these "posterior" estimates as resulting from a prior that must have been "uninformative" without attempting to find the explicit formulation of the prior that would lead to such a posterior? In general, within the realm of regression analysis, when is it okay to proceed along these lines (of treating the likelihood like a posterior) and when is it not okay? What about with frequentist methods that are not likelihood-based, such as quasi-likelihood methods, or ordinary or weighted least squares methods generally (where the coefficient estimates still have well-defined sampling distributions under the frequenist paradigm)?

Do the answers depend on whether my target of inference is coefficient point estimates, or the probability of a coefficient being within a particular range, or quantities of the predictive distribution?

$\endgroup$
4
$\begingroup$

This is basically a question about $p$-values and maximum likelihood. Let me quote Cohen (1994) in here

What we want to know is "Given this data what is the probability that $H_0$ is true?" But as most of us know, what it [$p$-value] tells us is "Given that $H_0$ is true, what is the probability of this (or more extreme) data?" These are not the same (...)

So $p$-value tells us what is the $P(D|H_0)$, while we are interested in $P(H_0|D)$ (see also the discussion on Fisherian vs Neyman-Pearson framework).

Let's forget for a moment about $p$-values. The probability of observing our data given some parameter $\theta$ is the likelihood function

$$ L(\theta | D) = P(D|\theta) $$

that is one way of looking at statistical inference. Another way is Bayesian approach where we want to learn directly (rather than indirectly) about $P(\theta|D)$ by employing the Bayes theorem and using priors for $\theta$

$$ \underbrace{P(\theta|D)}_\text{posterior} \propto \underbrace{P(D|\theta)}_\text{likelihood} \times \underbrace{P(\theta)}_\text{prior} $$

Now, if you look at the overall picture, you'll see that $p$-values and likelihood answer a different questions than Bayesian estimation.

So, while maximum likelihood estimates should be the same as MAP Bayesian estimates under uniform priors, you have to remember that they answer a different question.


Cohen, J. (1994). The earth is round (p<.05). American Psychologist, 49, 997-1003.

$\endgroup$
  • $\begingroup$ Thanks for your reply @Tim. I should have been clearer -- I understand that P(D|H) and P(H|D) are in general not the same, and that frequentists and Bayesians differ in opinion about whether it is appropriate to assign probability distributions to parameters (or hypotheses more generally). What I'm asking about is situations in which the (frequentist) sampling distribution of an estimator will be numerically equivalent to the (Bayesian) posterior distribution of the true parameter value. $\endgroup$ – Yakkanomica Nov 26 '15 at 0:07
  • $\begingroup$ Continuation of my previous comment: You wrote: "So, while maximum likelihood estimates should be the same as MAP Bayesian estimates under uniform priors," -- I'm asking if there are situations in which this relationship breaks down -- both in terms of the point estimates and the distributions around them. $\endgroup$ – Yakkanomica Nov 26 '15 at 0:10
  • $\begingroup$ One final addendum -- Some people would say that the main virtue of the Bayesian approach is it's ability to flexibly incorporate prior knowledge. To me, the appeal of the Bayesian approach is in interpretation -- the ability to assign a probability distribution to a parameter. Needing to specify priors is a nuisance. I want to know in what situations I can use frequentist methods but assign a Bayesian interpretation to the results by arguing that the frequentist and Bayesian results coincide numerically under plausibly noninformative priors. $\endgroup$ – Yakkanomica Nov 26 '15 at 0:22
  • 2
    $\begingroup$ @Yakkanomica I understand, it is an interesting question, but the simple answer (as stated above) is that you shouldn't make such interpretations because frequentest methods answer the different question than Bayesian. ML and MAP point estimates should agree, but the confidence intervals and HDI can differ and should not be interpreted interchangeability. $\endgroup$ – Tim Nov 26 '15 at 9:13
  • $\begingroup$ But @Tim, there are situations in which the confidence intervals and HDI do overlap. For example, compare the ML estimates on p.1906 with the Bayesian posterior estimates (based on uniform priors on the coefficients and IG prior on the scale) on p.1908: PROC GENMOD example. The ML point estimate and 95% confidence limits are very similar to the Bayesian posterior mean estimate and 95% HPD interval. $\endgroup$ – Yakkanomica Nov 26 '15 at 16:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.