It's about the demostration of the quadratic forms and chi-squared distribution.

Let's split the problem:

  1. We have a $n$ vector with n standardized normal distribution called $z={[z_1,z_2...z_n]}$. Obviously $z'z$ is a $\chi^2_n$
  2. If we have a symmetric, idempotent matrix (i.e. called $A$) then $z'Az \thicksim \chi^2_p $ where $p$ is the rank of $A$. I got this splitting the $A$ matrix with the spectral theorem into: $Q'\Lambda Q$ hence: $z'Az = z'Q'\Lambda Qz$ we call $w= Qz$ hence $z'Az =w'\Lambda w$. Because it's a idempotent matrix we have only 1s or 0s as eigenvalues and the number of 1's is exactly the rank of $A$. Then we have $w'w$ with with different length now as the number of the rank of $A$. We backward the process and we get: $z'Q'Qz$ but $Q'Q= I_{rank(A)}$ (they are orthogonal since we consider the eigenvectors) so we have the summation of $p$ $Z^2$.
  3. Now the question comes: if we have a symmetric, positive definite matrix $\Sigma_{nxn}$ my teacher told me that $z'\Sigma z$ is a $\chi^2_n $ distribution. I was trying to find a proof, but I can't find one.
  4. My efforts are: use the spectral theorem again, hence we have: $z'Q' \Lambda Qz$ but now $\Lambda$ is a diagonal matrix with real positive numbers (cause it's positive definite).

    $\Lambda$= \begin{bmatrix} \lambda_{1} & 0 & 0 & \dots & 0 \\ 0 & \lambda_{2} & 0 & \dots & 0 \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & 0 & \dots & \lambda_{n} \end{bmatrix}

We have called, as before, $w=Qz$ hence we have: $w'\Lambda w$. Let's suppose, that w is 3-dimensional vector with 3 elements called a,b and c. So $w=$ \begin{bmatrix} a \\ b \\ c \\ \end{bmatrix}

so $w'\Lambda w= a^2 \lambda_1+b^2 \lambda_2+c^2 \lambda_3$ And here I'm stuck. Maybe this last point is wrong. Therefore I'm asking for clarification and a clear proof of what my teacher said.


I found this:

Let $X$ be a $ K \times 1$ standard multivariate normal random vector, i.e., $X \thicksim N(0,I)$. Let $A$ be an orthogonal $K \times K$ real matrix. Define $Y=AX$ Then also $Y$ has a standard multivariate normal distribution, i.e., $Y \thicksim N(0,I)$. I'm okay with this, the proof is here: https://www.statlect.com/probability-distributions/normal-distribution-quadratic-forms So my problem would come something like this: $w'\Lambda w$ where $w=QZ$ and by this theorem $w \thicksim N(0,I)$ since Q is an orthogonal matrix. So, if w is a 3 dimensional vector I'll have something like this:

$\lambda_1 \chi^2_1 + \lambda_2 \chi^2_1 + \lambda_3 \chi^2_1 $ A lineare combination of chi-squared distribution with weights the value of the eigenvalues. Am I right? If I am, does this sum has a particular distribution?

  • 1
    $\begingroup$ Are you certain that in (3.), $z \sim \mathcal{N}(0,I)$ and not $z \sim \mathcal{N}(0,\Sigma^{-1})$? In which case clearly the quadratic is a $\chi^2$. Otherwise (for std. normal $z$ and symmetric PD $\Sigma$), I don't think $z'\Sigma z$ is distributed $\chi^2$: see answer (here)[stats.stackexchange.com/questions/9220/… for useful leads. $\endgroup$ – Nate Pope Oct 9 '17 at 2:30
  • $\begingroup$ I added a new part, I don't know if it will be a notification. I'm new here and I'll be glad if you can read my edit to the post. Thank you. $\endgroup$ – Mario Migliaccio Oct 9 '17 at 22:13
  • $\begingroup$ Take a look at the wikipedia page for the generalized $\chi^2$ distribution $\endgroup$ – Nate Pope Oct 9 '17 at 22:29
  • $\begingroup$ It has a chi-square distribution only under special conditions: stats.stackexchange.com/q/188626/119261. $\endgroup$ – StubbornAtom Jun 25 at 18:30

It is not true in general that $\mathbf{z}^\text{T} \mathbf{\Sigma} \mathbf{z} \sim \chi^2_p$ for any symmetric positive-definite (variance) matrix $\mathbf{\Sigma}$. Breaking this quadratic form down using the spectral theorem you get:

$$\begin{equation} \begin{aligned} \mathbf{z}^\text{T} \mathbf{\Sigma} \mathbf{z} = \mathbf{z}^\text{T} \mathbf{Q} \mathbf{\Lambda} \mathbf{Q}^\text{T} \mathbf{z} &= (\mathbf{Q}^\text{T} \mathbf{z})^\text{T} \mathbf{\Lambda} (\mathbf{Q}^\text{T} \mathbf{z}) \\[6pt] &= \sum_{i=1}^p \lambda_i ( \mathbf{q}_i \cdot \mathbf{z} )^2, \\[6pt] \end{aligned} \end{equation}$$

where $\mathbf{q}_1,...,\mathbf{q}_p$ are the eigenvectors of $\mathbf{\Sigma}$ (i.e., the columns of $\mathbf{Q}$). Define the random variables $y_i = \mathbf{q}_i \cdot \mathbf{z}$. Since $\mathbf{\Sigma}$ is a real symmetric matrix, the eigenvectors $\mathbf{q}_1,...,\mathbf{q}_p$ are orthonormal, which means that $y_1,...,y_p \sim \text{IID N}(0,1)$. Thus, we have:

$$\begin{equation} \begin{aligned} \mathbf{z}^\text{T} \mathbf{\Sigma} \mathbf{z} &= \sum_{i=1}^p \lambda_i ( \mathbf{q}_i \cdot \mathbf{z} )^2 \\[6pt] &= \sum_{i=1}^p \lambda_i \cdot y_i^2 \\[6pt] &\sim \sum_{i=1}^p \lambda_i \cdot \chi_1^2. \\[6pt] \end{aligned} \end{equation}$$

We can see that the distribution of the quadratic form is a weighted sum of $\chi_1^2$ random variables, where the weights are the eigenvalues of the variance matrix. In the special case where these eigenvalues are all one we do indeed obtain $\mathbf{z}^\text{T} \mathbf{\Sigma} \mathbf{z} \sim \chi_n^2$, but in general this result does not hold. Bodenham and Adams (2015) examine some approximations to this distribution, and provide comparisons with simulations.

| cite | improve this answer | |
  • $\begingroup$ Awesome, thanks! But, do we really need $\Sigma$ to be positive semi-definite for your proof? It seems to me that it's enough if it is symmetric. Then some $\lambda_i$ might be negative, but it still works. I did some calculations and it seems like allowing negative eigenvalues for $\Sigma$ works well! Thats not a proof, but I think your proof works with negative $\lambda_i$ as well. What do you think? $\endgroup$ – Curious Jun 25 at 14:10
  • $\begingroup$ That is correct; if it is not positive semi-definite then you get at least one negative eigenvalue and the quadratic form has a non-zero probability of being negative. However, in most statistical contexts you will encounter, the matrix $\mathbf{\Sigma}$ is a variance matrix. $\endgroup$ – Ben Jun 25 at 22:10

Regarding 3. above, suppose $\Sigma_{n\times n}$ is diagonal with diagonal elements $\{3, 2, 1\}$, then $\Sigma$ is symmetric positive definite. ...but $z'\Sigma z$ cannot be $\chi^2_n$ (chi-squared with 3 df), because $E\left[z'\Sigma z\right]=3E(z_1^2)+2E(z_2^2)+E(z_3^2)=6$, but the mean of a chi-squared with 3 df is 3.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.