At first glance, Pearson's chi-squared test seems flawed in a major way. Can you help me identify the error in my logic?

I have a multinomial distribution with $k$ outcomes, and $p_i$ denotes the hypothesized probability of outcome $i$. I obtain $n$ samples from this distribution, and let $n_i$ denote the number of times I observe outcome $i$ (so $\sum n_i=n$).

We have $E[n_i]=np_i$, $Var[n_i]=np_i(1-p_i)$ and $Cov[n_i,n_j]=-np_ip_j$ for $i\ne j$. Setting $x_i:=\frac{n_i-np_i}{\sqrt{np_i(1-p_i)}}$, we have $E[x_i]=0$, $Var[x_i]=1$ and $Cov[x_i,x_j]=-\sqrt{\frac{p_ip_j}{(1-p_i)(1-p_j)}}$.

Let $\vec{x}\in\mathbb{R}^k$ be the vector of $x_i$s, so $E[\vec{x}]=\vec{0}$ and $Var[\vec{x}]_{i,j}=Cov[x_i,x_j]$, which was calculated above. This covariance matrix has corank $1$ (this is easier to show before rescaling, and can include a proof in future edits if requested). If the $n_i$ are all sufficiently large, the distribution of each $x_i$ is roughly $\mathcal{N}(0,1)$. But $Var[\vec{x}]$ does not depend on $n$, and is really not that close to the identity matrix (especially if $k$ is small), so the $x_i$ are not uncorrelated and do not become any less correlated if $n$ is taken to be larger and larger.

And yet we estimate $\sum\limits_{i=1}^{k-1} x^2_i\sim \chi^2_{k-1}$ to test this hypothesis. The choice of not including $x^2_k$ is itself quite strange, because while it is true that $x_k$ completely depends on the other $x_i$, we've ignored their correlations in every other context! And isn't there an easy fix here?

Let $A$ be a $(k-1)\times k$ matrix with $ACov[\vec{x}]A^T=I_{k-1}$, the $(k-1)\times (k-1)$ identity matrix (which can be computed via e.g. SVD). Then $Var[A\vec{x}]=I_{k-1}$. As $n$ gets very large the distribution of the components of $A\vec{x}$ individually approach $\mathcal{N}(0,1)$, and their correlation matrix is the identity. This doesn't imply that their joint distribution approaches the joint gaussian distribution (as uncorrelated guassians are not necessarily independent), but I would imagine this is provably true.

And so why don't we test $\|A\vec{x}\|^2\sim \chi_{k-1}^2$ instead? This $A$ is not hard to compute as its complexity is in terms of $k$.


1 Answer 1


As I know the Pearson test uses:

$$X_i = \frac{n_i - np_i}{\sqrt{np_i}}$$ and not the $X_i$ that you mentioned. To prove the result of the Pearson test, let $\widehat{\boldsymbol{\theta}}_n = \frac1n \begin{pmatrix}n_1\\\vdots\\n_k\end{pmatrix}$ and $\boldsymbol{\theta} = \begin{pmatrix}p_1\\\vdots\\p_k\end{pmatrix}$ you can easily prove that

$$\sqrt{n} \left(\widehat{\boldsymbol{\theta}}_n - \boldsymbol{\theta}\right) \overset{d}\to \mathcal N\left(\boldsymbol{0}, \Sigma\right)$$

where $$\Sigma = \begin{bmatrix} p_1\left(1-p_1\right) & -p_1p_2 & \cdots & -p_1p_k\\ -p_1p_2 & p_2\left(1-p_2\right) & \cdots & -p_2p_k\\ \vdots & \vdots & \ddots & \vdots\\ -p_1p_n & -p_2p_k & \cdots & p_k\left(1-p_k\right) \end{bmatrix}$$

Let $\boldsymbol{A} = \begin{bmatrix} \frac1{\sqrt{p_1}} & \\ & \ddots \\ &&\frac1{\sqrt{p_k}}\end{bmatrix}$ then, $$\sqrt{n}\boldsymbol{A} \left(\widehat{\boldsymbol{\theta}}_n - \boldsymbol{\theta}\right) \overset{d}\to \mathcal N\left(\boldsymbol{0}, \boldsymbol{A}\Sigma\boldsymbol{A}^\intercal\right)$$

You can observe that $$\boldsymbol{X} = \sqrt{n}\boldsymbol{A} \left(\widehat{\boldsymbol{\theta}}_n - \boldsymbol{\theta}\right) $$

and if $\boldsymbol{u} = \begin{pmatrix}\sqrt{p_1}\\\vdots\\\sqrt{p_k}\end{pmatrix}$ then $$\boldsymbol{A}\Sigma\boldsymbol{A}^\intercal = \mathbf I - \boldsymbol{u}\boldsymbol{u}^\intercal$$


$$\boldsymbol{X} \overset{d}\to \mathcal N\left(\boldsymbol{0}, \mathbf I - \boldsymbol{u}\boldsymbol{u}^\intercal\right)$$ Since $\boldsymbol{u}^{\intercal} \boldsymbol{X}= 0$ then if $Z\sim \mathcal N(0, 1)$ then $$\boldsymbol{X} + Z\boldsymbol{u} \overset{d}\to \mathcal N(0, \mathbf I).$$

By Cochran's theorem, $$\left\|X\right\|^2 = \left\|\pi_{\boldsymbol{u}^\perp}\left(\boldsymbol{X} + Z\boldsymbol{u}\right)\right\|^2 \sim \chi^2_{k-1}$$

  • $\begingroup$ Beautiful; thank you. I suspect this implies that there is a very nice ``diagonal" $A$ satisfying the conditions of my question, to make it equivalent to the Pearson chi-square test $\endgroup$
    – Terence C
    Commented Feb 5 at 23:10
  • $\begingroup$ Probably. So your question has another test different from the one of Pearson? Is that right? $\endgroup$
    – Kroki
    Commented Feb 5 at 23:11
  • 1
    $\begingroup$ Based on your answer, I'm sure that the two are equivalent. $A$ will probably correct the $\frac{1}{\sqrt{1-p_i}}$ difference. $\endgroup$
    – Terence C
    Commented Feb 5 at 23:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.