What is a non-informative choice of parameters for a Dirichlet distribution?

Question

Dirichlet distribution is a conjugate prior for multinomial distribution. I want to impose a non-informative prior over sampling weights $\pi$ for a draw $x=(x_1,…,x_N)$ from a multinomial distribution with support $d=(d_1,…,d_K)$ (all the possible values that $x_i$ can take) and sampling weights $\pi=(\pi_1,…,\pi_K)$.

I was under impression that $Dir(\alpha)$ with $\alpha_i=1$ is a right choice. But I've read that (see e.g. this) $Dir(\alpha)$ with $\alpha_i=0$ yields an improper non-informative distribution.

Question:

1. Why $Dir(\alpha)$ with $\alpha_i=0$ is non-informative? Doesn't $\alpha\to 0$ impose higher sampling weight on a single data and zero on all others?
2. Shouldn't a uniform distribution $Dir(\alpha)$ with $\alpha_i=1$ be non-informative choice for the prior instead?

Answer 1

The main issue here is that "non-informative" is something of a term of art, and it can be formulated in various ways (see here for an interesting discussion on the subject). In a certain strict sense, there is no such thing as a "non-informative prior" since every prior distribution is a specific distribution that has a number of specific probabilistic implications. What we do have are several different methodologies that can form non-subjective priors (i.e., priors that depend only on the general form of the likelihood function without consideration of the data values).

There are several competing theories on formulating non-subjective priors. This includes the theory of "reference priors", the Jeffries priors, and various others. These theories lead to prior forms that are quite close to each other, but they do differ a little bit, and so there is also quite a bit of literature arguing over which is best. If you would like to learn more about this, I strongly recommend you read some of the works of José Bernardo, who is probably the preeminent Bayesian statistician in this field. (Another thing I would recommend is to read about the theory of "imprecise probability" by Peter Walley; in my view this method has a better claim to being truly objective and "non-informative" than choosing a specific prior via other theories.)

In regard to your specific questions, yes, the $\text{Dirichlet}(\mathbf{0})$ distribution is an improper distribution, so if you use it as a prior then it is an improper prior. As to whether this prior is better or worse than the flat prior, I will leave it to you to have a read of the literature on improper priors and see the advantages of each method. It is worth noting that they are not very different so long as you have a reasonable amount of data --- data manifests in the posterior as an increase of one in a parameter value for each observed data point. Bayesian analysis has a number of useful consistency theorems that establish that posterior beliefs converge even with different priors, and for priors like this, that are only slightly different, this convergence is quite rapid.

Answer 2

I was inclined to agree with you, because I know rstan uses αi=1 as the default Dirichlet prior choice. Their default choices are intended to be weakly informative. But I found this paper discussing why Dir(0) is a valid choice. I don't understand it well enough to give a good summary, but it looks like Dir(0) is the only choice which is noninformative under transformations that preserve normality.