# Why is there a change in the number of degrees of freedom when the following modification is made?

In the notes that I'm working through it says the following: "Let $$X_1,...,X_n$$ be a random sample from $$N(\mu,\sigma)$$

$$\sum^{n}_{i=1}\Bigg[\frac{(X_i-\mu)}{\sigma}\Bigg]^2$$ has a $$\chi^2$$ distribution with $$n$$ degrees of freedom.

Now if we modify this by replacing $$\mu$$ with $$\overline{X}$$ the distribution changes and we obtain: $$\sum^{n}_{i=1}\Bigg[\frac{(X_i-\overline{X})}{\sigma}\Bigg]^2$$ has a $$\chi^2$$ distribution with $$n-1$$ degrees of freedom."

My question is: why does the number of degrees of freedom change?

My understanding of what a degree of freedom is that the number of degrees of freedom is the number of values in the final calculation of a statistic that are free to vary. So surely as there are $$n$$ $$X_i$$ even when we introduce $$\overline{X}$$ the number of values that are free to change in the calculation of the statistic is still the same??

• Try it with $n = 2,$ Once you know $\bar X$ and $X_1,$ then you know $X_2.$ // The glib, supposedly intuitive, 'explanation' is that you "lose one degree of freedom estimating the mean." // More rigorously $\sum_i (X_i - \mu)^2$ can be decomposed into $\sum_i (X_i - \bar X)^2 +$ the square of one other normal random variable. // Some people are 'convinced' by a simulation of each and fitting the relevant CHISQ random variables with $n$ and $n-1$ DF, which I will attempt. – BruceET Apr 30 '19 at 1:45
• Consider that $\bar{X}$ is both closer to the data than $\mu$ is, and dependent on it. – Glen_b -Reinstate Monica Apr 30 '19 at 1:58
• The concept of degrees of freedom is thoroughly discussed in our thread at stats.stackexchange.com/questions/16921. – whuber Apr 30 '19 at 12:38

Suppose we have a random sample from $$\mathsf{Norm}(\mu, \sigma).$$

Let the $$V_1 = S^2 = \frac{1}{n-1}\sum_{i=1}^n (X_i - \bar X)^2$$ be the estimate of the population variance $$\sigma^2$$ when $$\mu$$ is unknown and estimated by $$\bar X.$$ Then $$Q_1 = \frac{(n-1)V_1}{\sigma^2} \sim \mathsf{Chisq}(n-1).$$

Let the $$V_2 = \frac{1}{n}\sum_{i=1}^n (X_i - \mu)^2$$ be the estimate of the population variance $$\sigma^2$$ when $$\mu$$ is known. Then $$Q_2 = \frac{nV_2}{\sigma^2} \sim \mathsf{Chisq}(n).$$

In the R code below, we choose $$m = 10^6$$ samples of size $$n = 5$$ from $$\mathsf{Norm}(\mu = 50, \sigma = 7).$$ Then we make histograms of $$Q_1$$ and $$Q_2.$$ Chi-squared densities with degrees of freedom $$4$$ and $$5,$$ respectively, fit the histograms. The density curve for $$\mathsf{Chisq}(5)$$ does not fit the histogram for $$Q_1.$$

set.seed(2019)
m = 10^6;  n = 5;  mu = 50;  sg = 7
x = rnorm(m*n, mu, sg)
MAT = matrix(x, nrow=m)      # each row a sample of             n
v1 = apply(MAT, 1, var)      # uses sample mean
v2 = rowSums((MAT - mu)^2)/n # uses population mean
q1 = (n-1)*v1 /sg^2
q2 = n*v2 / sg^2

mean(q1);  var(q1)
[1] 3.997226   # aprx E(Q1) = 4
[1] 8.00637    # aprx Var(Q1) = 8
mean(q2);  var(q2)
[1] 4.997005   # aprx E(Q2) = 5
[1] 9.98925    # aprx Var(Q2) = 10

par(mfrow=c(1,2))  # enables 2 panels per plot
hist(q1, prob=T, br=50, col="skyblue2", ylim=c(0,.2))
curve(dchisq(x, n), add=T, col="red", lwd=2, lty="dotted")
hist(q2, prob=T, br=50, col="skyblue2")
par(mfrow=c(1,1))


Your understanding of degrees of freedom is correct for one of the two senses of the term. The vector $$\left(X_1-\overline X,\ldots,X_n-\overline X\right)$$ has $$n-1$$ degrees of freedom because it is subject to the constraint that the sum of the components must be $$0,$$ so if you know $$n-1$$ of them plus that constraint, then you know all of them.

The other sense of the term degrees of freedom is used when one speaks of a chi-square distribution with a specified number of degrees of freedom. Consider any orthonormal basis of the $$(n-1)$$-dimensional space of $$n$$-tuples in which the sum of the components is $$0.$$ Let $$U_1,\ldots,U_{n-1}$$ be the scalar components of $$\left( X_1-\overline X, \ldots, X_n-\overline X \right)$$ with respect to that basis. Then \begin{align} & \left( X_1-\overline X\right)^2 + \cdots + \left( X_n - \overline X\right)^2 = U_1^2 + \cdots + U_{n-1}^2 \\[5pt] & \text{and } U_1,\ldots,U_{n-1} \sim \text{i.i.d. } N(0, \sigma^2). \end{align}

Your understanding of degrees of freedom is correct. The difference essentially boils down to a subtle difference between $$\mu$$ and $$\bar{X}$$.

The sample mean $$\bar{X}$$ is determined by the values of the observed samples. As a result, there's some redundancy between $$\bar{X}$$ and the individual values of $$x_i$$. Suppose we knew $$\bar{X}$$ and the first $$N-1$$ values. That's enough, because we can write out the equation for $$\bar{X}$$ as $$\bar{X}=\frac{1}{N}x_1 +\frac{1}{N}x_2 + \frac{1}{N}x_3+ \ldots + \frac{1}{N}x_{N-1} + \frac{1}{N}x_N$$

A little bit of rearrangement lets us find that missing value: $$x_N = \bar{X} - \frac{N-1}{N}(x_1+x_2+\ldots +x_{N-1})$$

This isn't the case when the population mean ($$\mu$$) is used, since it is not dependent on the observed data; it's known (or assumed to be known) ahead of time. You therefore need to use all $$N$$ values to produce calculate your test statistic.

• But isn't knowing the values of $(n-1)$ $x_i$'s and $\overline{X}$ the same thing as having to know n values, hence n degrees of freedom?? – stochasticmrfox May 17 '19 at 10:42