# Expectation of Fisher score not equal to 0 when parametrize Categorical distribution differently

Expectation of Fisher score should equal to zero. The prove can be found in many palces, such as wikipedia. But I tried a categorical distribution that is not parameterizatized minimally, the expected score is not zero. For example:

Case 1: Conanical form with natural parameters

In this case the parameterization is minimal. For $$x \sim Cat(\mathbf{\theta})$$ where $$\mathbf{\theta}= (\theta_1,\cdots,\theta_{K-1})^T,x\in 1:K$$, the density function is: \begin{align}P(x) &= \exp (\mathbf{\theta}^T \phi(x)-A(\theta)) \\ &\text{Where}\\ & \phi(x) = (I(x=1),\cdots,I(x=K-1))^T \\ & A(\mathbf{\theta}) = \log \left[1+\sum_{k=1}^{K-1}\exp(\theta_k)\right] \end{align} The corresponding score function is: $$S(\mathbf{\theta})_{: K-1 \times 1} = \begin{pmatrix}I(x=1) - \exp(\theta_1)/(1+\sum_{j=1}^{K-1}\exp(\theta_j))\\ \vdots \\ I(x=K-1) - \exp(\theta_{K-1})/(1+\sum_{j=1}^{K-1}\exp(\theta_j))\end{pmatrix}$$ The expectation is $$E_{x\sim Cat(\mathbf{\theta})} (S(\mathbf{\theta})) = \begin{pmatrix}0\\ \vdots \\ 0\end{pmatrix}$$ This is in accordance with the defintion. But I have different result when the parameterization is not minimal:

Case 2: Non-minimal parameterization

$$x \sim Cat(\mathbf{p})$$ where $$\mathbf{p}= (p_1,\cdots,p_K)^T,x\in 1:K$$ are the probabilities. This parameterization is not minimal for there is a linear correlation between the parameters: $$p_K = 1-\sum_{k=1}^{K-1}p_k$$. The density function is: $$P(x) = \exp (\sum_{k=1}^K I(x=k) \log p_k)$$ The corresponding score function is: $$S(\mathbf{p})_{: K\times 1} = \begin{pmatrix}I(x=1)/p_1\\ \vdots \\ I(x=K)/p_K\end{pmatrix}$$ The expectation is $$E_{x\sim Cat(\mathbf{p})} (S(\mathbf{p})) = \begin{pmatrix}1\\ \vdots \\ 1\end{pmatrix}$$ It's not zero.

Where did I do wrong? Is minimal parameterization a requirement for Fisher score? Or are there other requirements that I neglected?

• A compact parameter space is an assumption invoked for establishing a zero expectation for the score, if I remember correctly. Does that work in the second case with the parameter restriction? Commented Feb 10, 2023 at 13:04
• Thank you @ChristophHanck the first case is equivalent to applying restriction to the second case, so yes it does work, at least in categorical distributions. Is it a necesary condition for all Fisher score caculations? Commented Feb 10, 2023 at 13:21
• No, not necessary - while not exactly on topic of score identity, this is maybe useful_ en.wikipedia.org/wiki/Maximum_likelihood_estimation#Consistency Commented Feb 10, 2023 at 15:06
• Thank you @ChristophHanck it is very helpful! Commented Feb 13, 2023 at 9:18