I've read in many textbooks that the Pearson Chi-Square statistic is a score statistic in the multinomial setting (as well as others). I thought it would be a good exercise to derive this, but I am getting quite stuck.

Let $X\sim \text{Multinomial}(n,p)$. Consider the hypothesis test: $H_0:p=a$ vs $p\neq a$. The Pearson Chi-Square test is given by $$X^2 = \sum_{i=1}^k \dfrac{(x_i - na_i)^2}{na_i}$$ i.e. the classic "observed minus expected squared over expected" form. Now I attempt to derive the score statistic to show it is equal to the Pearson Chi-Square.


$$\begin{align*} L(p) &= n!\prod_{i=1}^k \dfrac{p_i^{x_i}}{x_i!}\\ \ell(p) &= \log{n!} + \sum_{i=1}^k x_i\log{p_i} + \sum_{i=1}^k \log{x_i!}\\ U(p) &= \dfrac{\partial\ell(p)}{\partial p} = \begin{pmatrix} \dfrac{x_1}{p_1}& \ldots & \dfrac{x_k}{p_k}\end{pmatrix}^\intercal \end{align*}$$

The MLE under the null is simply $\hat{p}_0 = a$. To get the MLE under the alternative hypothesis we can use Lagrange Multipliers (since the $p$ are constrained to sum to 1)

$$\begin{align*} \Lambda(p,\lambda) &= \ell(p) - \lambda\left(\sum_{i=1}^k p_i - 1 \right)\\ \mathbf{0} &\overset{set}{=} \dfrac{\partial\Lambda(p,\lambda)}{\partial (p,\lambda)^\intercal}\\ &= \begin{pmatrix} \dfrac{x_i}{p_i} -\lambda \\ \vdots\\ \sum_{i=1}^k p_i - 1 \end{pmatrix}\\ \end{align*}$$ so we have $\sum_{i=1}^k p_i =1$ and $p_i = x_i/\lambda$. Summing the latter from $i=1,\ldots,k$ we get $\lambda = \sum_{i=1}^k x_i = n$ and therefore the unconstrained MLE is given by $\hat{p}_i = x_i/n$.

The score statistic is given by $$U(\hat{p}_0)^\intercal I^{-1}(\hat{p}_0)U(\hat{p}_0)$$ where $I$ is the Fisher information matrix. Here is where I get a bit stuck. How do I get the Fisher Information matrix while taking into account the constraint on $p$? If I simply take the expected value of the negative second derivative of the log likelihood I have

$$I(p) = \text{E}\left[ \dfrac{-\partial^2 \ell(p)}{\partial p}\right] = \begin{pmatrix} n/p_1 & \ldots & 0 \\ & \ddots & \\ 0& & n/p_k\end{pmatrix}$$

but that doeesn't look quite right, (also I am not able to manipulate the score to look like the Pearson Chi-Square test either. I end up with an extra $n$). Some help would be appreciated.


I was recommended to reparametrize and include the constraint directly in the pdf. For simplicity lets use $k=3$ $$\begin{align*} \ell(p) &= \log{n!} + x_1\log{p_1} + x_2\log{p_2} + (n-x_1-x_2)\log{1-p_1-p_2} + \sum_{i=1}^k \log{x_i!}\\ U(p) &= \begin{pmatrix} \dfrac{x_1}{p_1} - \dfrac{n-x_1-x_2}{1-p_1-p_2} & \dfrac{x_2}{p_2} - \dfrac{n-x_1-x_2}{1-p_1-p_2} \end{pmatrix}^\intercal\\ I(p) &= \begin{pmatrix} \dfrac{n}{p_1} +\dfrac{n}{1-p_1-p_2} & \dfrac{n}{1-p_1-p_2} \\ \dfrac{n}{1-p_1-p_2} & \dfrac{n}{p_2} + \dfrac{n}{1-p_1-p_2} \end{pmatrix} \end{align*}$$

But when I tried to calculate the score $U(\hat{p}_0)^\intercal I^{-1}(\hat{p}_0)U(\hat{p}_0)$ I am stuck in algebra nightmare, and after several hours haven't gotten anywhere close to Pearson's Chi-Square.

  • $\begingroup$ I do not think your Fisher Information is correct. It's definitely not diagonal. $\endgroup$ – JohnK Jan 23 '16 at 1:09
  • $\begingroup$ I agree, I haven't figured it out. $\endgroup$ – bdeonovic Jan 23 '16 at 1:10
  • $\begingroup$ Also, could you please provide a reference for the claim that these two statistics are equal? $\endgroup$ – JohnK Jan 23 '16 at 1:17
  • $\begingroup$ colorado.edu/economics/morey/7818/estimation/… $\endgroup$ – bdeonovic Jan 24 '16 at 17:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.