15
$\begingroup$

This is a very basic question. Why do we use a chi square distribution? What is the meaning of this distribution? Why is this the distribution used for creating a confidence interval for the variance?

Every place I google for an explanation just presents this a fact, explaining when to use chi, but not explaining why to use chi, and why it look the way it does.

Many thanks to anyone that can point me towards the right direction and that is - really understanding why I am using chi when I am creating a confidence interval for the variance.

$\endgroup$
  • 4
    $\begingroup$ You use it because - when the data are normal - $Q = (n-1)\frac{s^2}{\sigma^2}\sim \chi^2_{n-1}$. (This makes $Q$ a pivotal quantity) $\endgroup$ – Glen_b Nov 13 '13 at 12:34
  • 2
    $\begingroup$ See also stats.stackexchange.com/questions/15711/… and its links. $\endgroup$ – Nick Cox Nov 13 '13 at 12:45
  • 1
    $\begingroup$ For those who are interested in the applications of or further research into $\chi^2$, you will want to pay attention to the distinction between a $\chi^2$ ("chi-squared") distribution and a $\chi$ ("chi") distribution (it is the square root of a $\chi^2$, unsurprisingly). $\endgroup$ – whuber Nov 13 '13 at 14:53
23
$\begingroup$

Quick answer

The reason is because, assuming the data are i.i.d. and $X_i\sim N(\mu,\sigma^2)$, and defining \begin{eqnarray*} \bar{X}&=&\sum^N \frac{X_i}{N}\\ S^2 &=& \sum^{N} \frac{(\bar{X}-X_i)^2}{N-1} \end{eqnarray*} when forming confidence intervals, the sampling distribution associated with the sample variance ($S^2$, remember, a random variable!) is a chi-square distribution ($S^2(N-1)/\sigma^2 \sim \chi^2_{n-1}$), just as the sampling distribution associated with the sample mean is a standard normal distribution ($(\bar{X}-\mu)\sqrt{n}/\sigma \sim Z(0,1)$) when you know the variance, and with a t-student when you don't ($(\bar{X}-\mu)\sqrt{n}/S \sim T_{n-1}$).

Long answer

First of all, we'll prove that $S^2(N-1)/\sigma^2$ follows a chi-square distribution with $N-1$ degrees of freedom. After that, we'll see how this proof is useful when deriving the confidence intervals for the variance, and how the chi-square distribution appears (and why it is so useful!). Let's begin.

The proof

For this, maybe you must get used to the chi-square distribution in this Wikipedia article. This distribution has only one parameter: the degrees of freedom, $\nu$, and happens to have a Moment Generating Function (MGF) given by: \begin{equation*} m_{\chi^2_\nu}(t)=(1-2t)^{-\nu/2}. \end{equation*} If we can show that the distribution of $S^2(N-1)/\sigma^2$ has a moment generating function like this one, but with $\nu=N-1$, then we have shown that $S^2(N-1)/\sigma^2$ follows a chi-square distribution with $N-1$ degrees of freedom. In order to show this, note two facts:

  1. If we define, \begin{equation*} Y = \sum \frac{(X_i-\bar{X})^2}{\sigma^2} = \sum Z_i^2, \end{equation*} where $Z_i\sim N(0,1)$, i.e., standard normal random variables, the moment generating function of $Y$ is given by \begin{eqnarray*} m_Y(t) &=& \mathbb{E}[e^{tY}]\\ &=&\mathbb{E}[e^{tZ_1^2}]\times \mathbb{E}[e^{tZ_2^2}]\times ...\mathbb{E}[e^{tZ_N^2}]\\ &=&m_{Z_i^2}(t)\times m_{Z_2^2}(t)\times ...m_{Z_N^2}(t). \end{eqnarray*} The MGF of $Z^2$ is given by \begin{eqnarray*} m_{Z^2}(t) &=& \int_{-\infty}^{\infty} f(z)\exp(tz^2)dz\\ &=&(1-2t)^{-1/2}, \end{eqnarray*} where I have used the PDF of the standard normal, $f(z)=e^{-z^2/2}/\sqrt{2\pi}$ and, hence, \begin{equation*} m_Y(t)=(1-2t)^{-N/2}, \end{equation*} which implies that $Y$ follows a chi-square distribution with $N$ degrees of freedom.

  2. If $Y_1$ and $Y_2$ are independent and each distribute as a chi-square distribution but with $\nu_1$ and $\nu_2$ degrees of freedom, then $W=Y_1+Y_2$ distributes with a chi-square distribution with $\nu_1+\nu_2$ degrees of freedom (this follows from taking the MGF of $W$; do this!).

With the above facts, note that if you multiply the sample variance by $N-1$, you obtain (after some algebra), \begin{equation*} (N-1)S^2 = -n(\bar{X}-\mu)+\sum(X_i-\mu)^2, \end{equation*} and, hence, dividing by $\sigma^2$, \begin{equation*} \frac{(N-1)S^2}{\sigma^2}+\frac{(\bar{X}-\mu)^2}{\sigma^2/N}=\sum \frac{(X_i-\mu)^2}{\sigma^2}. \end{equation*} Note that the second term in the left-side of this sum distributes as a chi-square distribution with 1 degree of freedom, and the right-hand side sum distributes as a chi-square with $N$ degrees of freedom. Therefore, $S^2(N-1)/\sigma^2$ distributes as a chi-square with $N-1$ degrees of freedom.

Calculating the Confidence Interval for the variance.

When looking for a confidence interval for the variance, you want to know the limits $L_1$ and $L_2$ in \begin{equation*} \mathbb{P}\left(L_1\leq \sigma^2 \leq L_2\right) = 1-\alpha. \end{equation*} Let's play with the inequality inside the parenthesis. First, divide by $S^2(N-1)$, \begin{equation*} \frac{L_1}{S^2(N-1)}\leq \frac{\sigma^2}{S^2(N-1)} \leq \frac{L_2}{S^2(N-1)}. \end{equation*} And then remember two things: (1) the statistic $S^2(N-1)/\sigma^2$ has a chi-squared distribution with $N-1$ degrees of freedom and (2) the variances is always greather than zero, which implies that you can invert the inequalities, because\begin{eqnarray*} \frac{L_1}{S^2(N-1)}\leq \frac{\sigma^2}{S^2(N-1)} &\Rightarrow& \frac{S^2(N-1)}{\sigma^2}\leq \frac{S^2(N-1)}{L_1},\\ \frac{\sigma^2}{S^2(N-1)} \leq \frac{L_2}{S^2(N-1)} &\Rightarrow& \frac{S^2(N-1)}{L_2} \leq \frac{S^2(N-1)}{\sigma^2},\\ \end{eqnarray*} hence, the probability we are looking for is: \begin{equation*} \mathbb{P}\left(\frac{S^2(N-1)}{L_2} \leq \frac{S^2(N-1)}{\sigma^2}\leq \frac{S^2(N-1)}{L_1}\right) = 1-\alpha. \end{equation*} Note that $S^2(N-1)/\sigma^2 \sim \chi^2(N-1)$. We want then, \begin{eqnarray*} \int_{\frac{S^2(N-1)}{L_2}}^{N-1}p_{\chi^2}(x)dx &=& (1-\alpha)/2\ \ \ ,\\ \int_{N-1}^{\frac{S^2(N-1)}{L_1}}p_{\chi^2}(x)dx &=& (1-\alpha)/2\ \ \, \end{eqnarray*} (we integrate up to $N-1$ because the expected value of a chi-squared random variable with $N-1$ degrees of freedom is $N-1$) or, equivalently, \begin{eqnarray*} \int_{0}^{\frac{S^2(N-1)}{L_2}}p_{\chi^2}(x)dx=\alpha/2,\\ \int_{\frac{S^2(N-1)}{L_1}}^{\infty}p_{\chi^2}(x)dx=\alpha/2. \end{eqnarray*} Calling $\chi^2_{\alpha/2}=\frac{S^2(N-1)}{L_2}$ and $\chi^2_{1-\alpha/2}= \frac{S^2(N-1)}{L_1}$, where the values $\chi^2_{\alpha/2}$ and $\chi^2_{1-\alpha/2}$ can be found in chi-square tables (in computers mainly!) and solving for $L_1$ and $L_2$, \begin{eqnarray*} L_1 &=& \frac{S^2(N-1)}{\chi^2_{1-\alpha/2}},\\ L_2 &=& \frac{S^2(N-1)}{\chi^2_{\alpha/2}}. \end{eqnarray*} Hence, your confidence interval for the variance is \begin{equation*} C.I.=\left(\frac{S^2(N-1)}{\chi^2_{1-\alpha/2}}, \frac{S^2(N-1)}{\chi^2_{\alpha/2}}\right). \end{equation*}

$\endgroup$
  • 1
    $\begingroup$ Simply because $S^2$ does not follow a centered chi-square distribution, while $S^2(N-1)/\sigma^2$ does and, therefore, its easier to work with. Are you asking for a derivation for that? (i.e., you want someone to show you that $S^2(N-1)/\sigma^2$ follows a chi-square distribution with $N-1$ degrees of freedom?) $\endgroup$ – Néstor Nov 13 '13 at 14:16
  • 4
    $\begingroup$ It would be helpful to modify this answer to include the very strong but unstated assumption that the sample variance follows a chi-squared distribution when the underlying data are independent and follow a normal distribution. Unlike the theory of the distribution of the sample mean, where in practice its sampling distribution will be approximately Normal to reasonable accuracy in many situations, this same asymptotic behavior tends not to happen with the sample variance (until sample sizes become extremely large). $\endgroup$ – whuber Nov 13 '13 at 14:58
  • 1
    $\begingroup$ Oops. So, so true! This actually came from a problem solution that I handed out to some students, where I state on the question all these assumptions. I edited the answer now. $\endgroup$ – Néstor Nov 13 '13 at 15:51
  • 1
    $\begingroup$ @user34756 The reason we don't use the distribution of $S^2$ directly is that its distribution depends on the value of a parameter. You may find it useful to investigate the use of pivotal quantities in constructing confidence intervals. $\endgroup$ – Glen_b Nov 14 '13 at 0:15
  • 1
    $\begingroup$ Isn't $f(z) = e^{-z^2/2}$ instead of $f(z) = e^{-z^2}$ ? $\endgroup$ – Benoît Legat Nov 15 '14 at 14:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.