What is the interpretation for the priors in the derivation of Laplace smoothing? [duplicate]

Question

Laplace smoothing has a generalisation that can be justified with the use of Bayes formula. Let $f(x;\alpha,\beta)$ be the (non-normalised) beta distribution, i.e. $$f(x;\alpha,\beta) = x^{\alpha-1}(1-x)^{\beta-1},\qquad\alpha,\beta\in\mathbb R^+.$$ Note that we can still consider $f(x;0,0)$ and we will, but this is not normalisable. In all the other cases, the normalisation constant is the Beta function $B(\alpha,\beta)$. Let $\pi$ be a probability parameter that we want to estimate and assume that we have $N$ conditionally independent trials of the same binary random variable $x$. We then have $$p(x|\pi) = \prod_{i=1}^N\pi^{x_i}(1-\pi)^{1-x_i} = \pi^s(1-\pi)^{n-s}$$ where $$s = \sum_{i=1}^Nx_i.$$ For the prior probability we choose $p(\pi;\alpha) = f(\pi;\alpha,\alpha)$, which is improper for $\alpha = 0$. The posterior probability is thus given by $$p(\pi|x;\alpha) = \frac{\pi^{s+\alpha-1}(1-\pi)^{n-s+\alpha-1}}{B(s+\alpha, n-s+\alpha)}.$$ If we now take the conditional expectation of $\pi$ as an estimator for $\pi$, we then get to $$E_{\pi|x;\alpha}[\pi] = \frac{s+\alpha}{n+2\alpha}.$$ We then see that, with $\alpha = 1$ we recover the Laplace smoothing. In this case, the prior $p(\pi;\alpha)$ is uniform. The case $\alpha = 0$ is the result that one would expect without Laplace smoothing. However, the prior is improper in this case.

The question is then: what is the interpretation for the priors? In the case $\alpha = 1$ we get a uniform distribution, meaning that every value of $\pi$ is equally probable. In fact, it doesn't even matter if we then define the prior to vanish on 0 and 1, we would still get Laplace smoothing. Given that we know nothing about $\pi$, any choice is as good as any, but I guess the uniformity hypothesis is quite a natural one.

The case $\alpha = 0$ is even more interesting, since the prior is improper. Whilst not a probability density, one can still look at its shape and conclude that we are modelling $\pi$ to either be 0 or 1 with high probability, and any intermediate value with lower probability. This is not quite like saying that we allow for the impossible and certain cases as well as the others, because in fact we are expressing a propensity towards these two edge cases rather than anything in between.

Answer 1

Using the more traditional $\alpha,\beta$ notation rather than $\alpha,n-s+\alpha$, anytime $\alpha+\beta<1$ you will have an improper prior. There are three primary uninformative conjugate priors for the binomial distribution. They are the Haldane, the Jeffreys, and the Uniform priors. None of them are uninformative, they are just weakly informative.

The Haldane, $(0,0)$, places infinite weight on zero and one and minimal weight on $p=.5$. The MAP estimator equals the MLE when $(\alpha,\beta)=(0,0).$ Its origin is to test processes whose sample size would be one and where either a zero or a one would be the desired answer. I believe the canonical example for this is testing whether or not a substance is soluble in water. If you put it in water and it dissolves, then you also destroy the substance. I believe the proper interpretation of the Haldane prior is that you have no idea what the true parameter is, but you will accept the outcome of any data as the best answer.

The Jeffreys prior $\left(\frac{1}{2},\frac{1}{2}\right)$ also grants infinite weight to zero and one, but the results are invariant to the monotone transformation of the parameters. It also links to Frequentist thinking because it is based on the expectation of Fisher Information and so depends, in part, on the sample space. It guarantees at least some weight to both cases and so rules out double-sided coins.

The Uniform prior could be interpreted as all choices having an equal opportunity, except zero and one. It excludes double-headed coins, except at the limit where $\lim_{n\to\infty}$. It also has an expected value of $\mathbb{E}(p)=.5$ and so biases it toward the center.

I would also like to include the triangular distribution as a weakly informative prior, though you didn't mention it. It is $(\alpha,\beta)\in\{(1,2),(2,1)\}$. Its interpretation is either that $\theta$ is more likely to the left than the right, or in the alternative case, more likely to the right than the left. Given any point estimate, in the absence of data, if you had to gamble, then you would tend to gamble left over right or right over left. It is a little bit more information than the uniform, but not much.