Is assuming a vague prior different than assuming nothing?

Question

If you use a vague prior in a Bayesian analysis so that $\Pr(A) = 1\mathbin/n$, then

\[ \Pr(\pi\mid x) = \frac{\Pr(x\mid \pi)\Pr(\pi)}{\Pr(x)} = \frac{\mathcal{L}(\pi;x)}{n\Pr(x)} \propto \mathcal{L}(\pi;x) \]

Which seems like it's what we would get if we just ignored priors altogether, computed a likelihood and normalized it to a probability.

So is assuming a vague prior like assuming nothing? I guess assuming a vague prior is like assuming you can infer a probability distribution from a likelihood. But what are you really assuming in that case?

It seems tempting to conclude that a vague prior is like assuming nothing, as this lifts one of the "burdens" of Bayesian analysis, but I remember being told that vague priors are not really the same as assuming nothing.

edit: I thought about this more, and I guess assuming a likelihood can just be normalized to a probability is pretty presumptuous, and it becomes clearer if you choose an example where the prior is very obviously not uniform.

Eg: I have a machine that can tells me whether or not sun has exploded with a 5% chance of error. One day it tells me the sun has exploded. Has it really?

P(machine says exploded | sun exploded) = 0.95, P(machine says exploded | sun hasn't exploded) = 0.05, but it's a bit of a leap to say that there is thus a 95% chance that the sun exploded.

Answer 1

Fast answer: there are many ways to get a vague prior. A 'nothing' prior as you mentioned, however, might seem vague, but actually might not exist. Coming up with vague priors is harder than you might think...

Details:

If you 'assume nothing', as you have defined it, you are saying that the posterior distribution is proportional to the likelihood alone. This basically means that the prior is 'flat'. If your parameter is discrete and finite, or continuous with finite support, you are ok, because the 'assume nothing' prior is a perfectly valid uniform prior and would often be called a vague prior.

Thus, for $X\sim Bin(\theta,n=10)$ you can have an 'assume nothing' prior that is $\theta\sim Unif(0,1)$. This would often be a good choice for a vague prior, which is usually just meant to be a prior that reflects little prior knowledge about the parameter, or, in particular, is easily data-dominated so that switching from one vague prior to another vague prior should give basically the same posterior distribution.

However, you often can't assume nothing and get a vague prior. For instance, if you consider the model $Y\sim Poisson(\lambda)$, the 'flat' prior is one that give equal probability for all $0\le \lambda<\infty$. Unfortunately there is no valid distribution that is flat from zero to infinity so it can't be a vague prior. When an 'illegal' prior like this is used (i.e. the posterior distribution is proportional to the likelihood), we call these 'improper priors'. Occasionally these fail spectacularly, because using a non-valid prior might lead to a non-valid (in the sense that it can't be normalized to give a density) posterior.

Another problem with flat priors is they are only flat under some parameterizations. For instance, if we observe a Binomial random variable, I might write the problem as $Y\sim Binomial(p,n=10)$ and use the flat prior on $p$: $p\sim Unif(0,1)$, while you might think of the problem as $Y\sim Binomial(\phi = p/(1-p),n=10)$ and put a flat prior on the odds $\phi$. It turns out that a flat prior on $p$ isn't the same as a flat prior on the odds $\phi$, so the concept of a flat prior is dependent on the parameterization of the problem (an attempt to make vague priors that are invariant to change of parameter leads to what are called 'Jeffrey's Priors').

In general a vague prior is one that is fairly flat over a very large range of your parameter, so that it doesn't have much influence on the posterior. A common choice of vague prior for parameters defined from $-\infty$ to $\infty$, for example, is a normal distribution with an exceedingly large variance.