2
$\begingroup$

The improper uniform distribution for parameter $\theta$ is :

$p(\theta)=1,\ for -\infty<\theta<\infty$.

It is called "improper" since it does not integrate to 1. Because Bayesian theorem is : $p(\theta|y)\propto L(\theta;y)p(\theta)\tag{1}$ when we use improper uniform distribution (i.e., $p(\theta)=1$), then the Equation (1) becomes: $p(\theta|y)\propto L(\theta;y)\tag{2}$ I think Equation (2) means that: when using "improper uniform priors" in Bayesian analysis, it is equal to the maximum likelihood estimates (MLE).

My question is:

(1) Am I right?

(2) If right, what is the advantage by using Bayesian technique if it is the same as MLE? Because I find in some situations, the "uniform" prior distribution is used widely such as Example 6.1.1 in this book (i.e., ... ~ dflat()). I attached this example below for better illustration.

(3) But the value of sigma2 is not the same (71.3 in MLE, but 149.8 in WinBUGS for Bayesian). Why does it happen, while alpha and beta are the same in these two methods?

Appendix:

Gelfand et al. (1990, p.978) examine growth data from 30 young rats whose weights were measured weekly for five weeks. In this example we fit a linear regression to the 9th rat's data. The response variable $y_{i},\ i=1,...,5$ is the weight, in grams, on day $x_{i}$. \begin{align} Y_{ij} &\sim \mathcal N(\alpha_i+\beta_i x_{ij},\sigma^2_c)\qquad i=1,\ldots,k\ j=1,\ldots,n\\ \left(\begin{matrix}\alpha_i\\\beta_i\end{matrix}\right)&\sim \mathcal N\left(\left(\begin{matrix}\alpha_c\\\beta_c\end{matrix}\right),\Sigma_c \right)\qquad i=1,\ldots,k\\ \mu_c &\sim \mathcal N(\mu,C)\\ \Sigma_c &\sim \mathcal W((\rho R)^{-1},\rho)\\ \sigma_c^2 &\sim \mathcal {IG}(\nu_0/2,\nu_0\tau^2_0/2) \end{align}

They specify improper uniform priors for all parameters, and so the posterior mode will be equal to the maximum likelihood estimates: $\alpha = 284.8$, $\beta = 7.31$, $\sigma^{2} = 71.3$.

The WinBUGS code is:

model{
    for (i in 1:5) {
        y[i] ~ dnorm(mu[i], tau)
        mu[i] <- alpha + beta*(x[i] - mean(x[]))   # center covariates
    }
    # Jeffreys priors
    alpha ~ dflat()
    beta ~ dflat()
    tau <- 1/sigma2
    log(sigma2) <- 2*log.sigma
    log.sigma ~ dflat()
}

# data
list(y=c(177,236,285,350,376),
     x=c(8,15,22,29,36))

# initial data
# In the book, there is no initial data, but the model will fall when "gen inits"
list(alpha = 0, beta = 0, log.sigma = 10)

I check the values of the maximum likelihood estimates using R, and I think the values in the book is right (i.e., 284.8, 7.31, 71.3).

y <- c(177, 236, 285, 350, 376)
x <- c(8, 15, 22, 29, 36)

# Likelihood
mnf <- function(pa,data){
  mu <- pa[1] + pa[2]*(x - mean(x))     
  pdf <- dnorm(data, mu, sqrt(pa[3]))   # pdf: y's probability distribution
  l = sum(log(pdf))   
  return(-l)      # maximum
}

ML.growth = nlminb(start = c(250, 5, 10),   # pa[1] ~ pa[3] initial value
                  objective = mnf,
                  data = y,
                  lower = c(-Inf,-Inf,0),   
                  upper = c(Inf,Inf,Inf))
ML.growth$par

The paper of Gelfand et al. (1990) can be found here. The example is in Section 6 "A Hierarchical Model", whose data is given in Table 3. $\alpha$ means the 9th rat's weight on the mean of the whole duration (22 day in this example); $\beta$ means the unit increase of weight when passing one day.

$\endgroup$
  • $\begingroup$ Please make the question self-contained by writing the model of Gelfand et al. (1990). Otherwise, $\alpha$ and $\beta$ are meaningless. $\endgroup$ – Xi'an Jul 6 at 9:46
  • $\begingroup$ @Xi'an Thanks, I have added the information about Gelfand et al. (1990) at the end of my post. $\endgroup$ – T X Jul 7 at 3:32
  • $\begingroup$ That does not make the question self-contained since readers need to follow the link to understand the model. $\endgroup$ – Xi'an Jul 7 at 8:52
10
$\begingroup$

In a rather narrow sense, you are correct (1): if $$\pi(\theta|x) \propto L(\theta|x)$$ then $$\underbrace{\arg \max_\theta \pi(\theta|x)}_\text{MAP estimate} = \underbrace{\arg \max_\theta L(\theta|x)}_\text{ML estimate}$$ However, this coincidence is not of considerable interest as:

  1. [Re. (1) and (3):] It is not invariant by repameterisation, i.e., the flat prior is only flat for one pameterisation. If one considers $\xi=h(\theta)$ as the new pameterisation of the model, the prior of $\xi$ is no longer flat and the MAP then differs from the MLE. In your example, the prior on $\sigma^2$ is $\sigma^{-2}$ not a
  2. [Re. (2):] The purpose of Bayesian analysis is not to return point estimates but a whole distribution on the parameter, conditional on the data. The posterior distribution provides optimal decisions (based on a loss function) and uncertainty quantification.
  3. [Re. (2):] The MAP estimator is fringe Bayesian in that there does not exist a conventional loss function that returns the MAP as the optimal decision. It further depends on the choice of the dominating measure.
  4. [Re. (3):] My almost prehistorical understanding of BUGS (i.e., circa 1992) is not accepting flat priors as far as I know.
  5. [Re. (3):] Regarding the R code, the function
mnf <- function(pa,data){
mu <- pa[1] + pa[2]*(x - mean(x))   
pdf <- dnorm(data, mu, sqrt(pa[3]))
l = sum(log(pdf))   
return(-l)}

returns the log-likelihood, not the maximum of the likelihood.

$\endgroup$
  • $\begingroup$ Could you explaine a bit more about your Answer 1 (which is about my Question 3)? I suppose what you mean is about the change of density distribution due to giving a function to the variable. In post here, the author used improper uniform distribution for log.sigma. He said the posterior mode will be equal to the maximum likelihood estimates: $\hat{\sigma^{2}}$ = 71.3. But the posterior mode of ${\sigma^{2}}$ is 149.8 in my computer, or 145.6 in the book - NOT equal. Is the book wrong? $\endgroup$ – T X Jul 7 at 3:44
  • $\begingroup$ The maximum likelihood for $\log(\sigma)$ is the logarithm of the maximum likelihood for $\sigma$ but this invariance does not hold for the posterior mode. $\endgroup$ – Xi'an Jul 7 at 8:54
  • $\begingroup$ I was still confused (sorry...), the authors said "They specify improper uniform priors for all parameters, so the posterior mode will be equal to the maximum likelihood estimates". However, $\sigma$ is not "equal" (71.3 V.S. 149.8), though $\alpha$ and $\beta$ are equal. Is the word "all" wrong? Because it didn't contain $\sigma$. (Your edit is amazing! I think I can not write those equations like you for a very long time) $\endgroup$ – T X Jul 8 at 12:33
  • $\begingroup$ There seems to be a clash between the JASA paper and this description with "uniform priors for all parameters". $\endgroup$ – Xi'an Jul 8 at 13:14
2
$\begingroup$

The whole business of improper priors in Bayesian analysis arises because some users of Bayesian reasoning believe that proper priors tend to be derived from 'subjective' opinions and thus would taint the Bayesian inference with subjectivity. It turns out that ignoring the impropriety of the priors can nevertheless yield usable results.

There are other schools of thought depending on one's interpretation of what is meant by probability. One train of thought is that everything in statistics is more or less subjective, not just the priors in Bayesian analysis. All models are approximations; models are chosen subjectively. If you think on those lines, then you will choose uninformative priors only if it is really true that you know absolutely nothing about the variable in question. But if you do know something - and quite often it will turn out that you do know something - then you should use informative priors. They will never be improper.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.