There is a catalog of noninformative priors over here:
http://www.stats.org.uk/priors/noninformative/YangBerger1998.pdf
in page 11, they give the noninformative Jeffreys prior for the Dirichlet distribution. They give the Fisher information matrix for the Dirichlet. Can someone tell me exactly what is cell (i,j) there for the matrix?
Is it all 0s, except for the diagonals and the upper right element and the bottom left element?
Thanks.
Let's work it out.
The logarithm of the Dirichlet density function is
$$\lambda(\mathbf{x}|\mathbf{\alpha}) = \log(\Gamma(\alpha_0)) - \sum_{i=1}^{k}{\log(\Gamma(\alpha_i)))} + \sum_{i=1}^{k}{(\alpha_i - 1)\log(x_i)},$$
where $\alpha_0 = \alpha_1 + \alpha_2 + \cdots + \alpha_k$.
Taking second partial derivatives with respect to the parameters $\alpha_i$ is particularly simple; all we really need to know (in addition to the most basic properties of derivatives) is that $\partial \alpha_0 / \partial \alpha_i = 1$ and $\partial x_j / \partial \alpha_i = 0$. Thus
$$\frac{\partial \lambda}{\partial \alpha_i} = \psi(\alpha_0) - \psi(\alpha_i) + \log(x_i)$$
and
$$\frac{\partial^2 \lambda}{\partial \alpha_i \partial \alpha_j} = \psi'(\alpha_0) - \psi'(\alpha_j)\delta_{i j},$$
where $\psi$ (the digamma function) is the derivative of $\log(\Gamma)$ and $\delta_{i j} = 1$ if and only if $i = j$ and is $0$ otherwise: that is, it's the $k$ by $k$ identity matrix. The Fisher Information Matrix is, by definition, the negative expectation of the matrix of second partial derivatives. Because its entries are constant with respect to the random variable $\mathbf{x}$, taking expectations is trivial. We obtain a matrix with the values $\psi'(\alpha_i)$ along the diagonal and $\psi'(\alpha_0)$ is subtracted everywhere, showing that @onestop's interpretation is correct. ("$PG(1,\alpha)$" is merely an idiosyncratic notation for the polygamma function $\psi'(\alpha)$.)
I think it's meant to have constant off-diagonal entries, i.e. it could also be written $$I(\alpha_1, \alpha_2, \ldots, \alpha_k) = \operatorname{diag}\left[ PG(1,\alpha_1) , PG(1, \alpha_2), \ldots, PG(1,\alpha_k) \right] - PG(1,\alpha_0) J_k $$ where $J_k$ is a $k \times k$ matrix of ones.
