Why is there a change in the number of degrees of freedom when the following modification is made?

Question

In the notes that I'm working through it says the following: "Let $X_1,...,X_n$ be a random sample from $N(\mu,\sigma)$

$$\sum^{n}_{i=1}\Bigg[\frac{(X_i-\mu)}{\sigma}\Bigg]^2$$ has a $\chi^2$ distribution with $n$ degrees of freedom.

Now if we modify this by replacing $\mu$ with $\overline{X}$ the distribution changes and we obtain: $$\sum^{n}_{i=1}\Bigg[\frac{(X_i-\overline{X})}{\sigma}\Bigg]^2$$ has a $\chi^2$ distribution with $n-1$ degrees of freedom."

My question is: why does the number of degrees of freedom change?

My understanding of what a degree of freedom is that the number of degrees of freedom is the number of values in the final calculation of a statistic that are free to vary. So surely as there are $n$ $X_i$ even when we introduce $\overline{X}$ the number of values that are free to change in the calculation of the statistic is still the same??

Answer 1

Suppose we have a random sample from $\mathsf{Norm}(\mu, \sigma).$

Let the $V_1 = S^2 = \frac{1}{n-1}\sum_{i=1}^n (X_i - \bar X)^2$ be the estimate of the population variance $\sigma^2$ when $\mu$ is unknown and estimated by $\bar X.$ Then $Q_1 = \frac{(n-1)V_1}{\sigma^2} \sim \mathsf{Chisq}(n-1).$

Let the $V_2 = \frac{1}{n}\sum_{i=1}^n (X_i - \mu)^2$ be the estimate of the population variance $\sigma^2$ when $\mu$ is known. Then $Q_2 = \frac{nV_2}{\sigma^2} \sim \mathsf{Chisq}(n).$

In the R code below, we choose $m = 10^6$ samples of size $n = 5$ from $\mathsf{Norm}(\mu = 50, \sigma = 7).$ Then we make histograms of $Q_1$ and $Q_2.$ Chi-squared densities with degrees of freedom $4$ and $5,$ respectively, fit the histograms. The density curve for $\mathsf{Chisq}(5)$ does not fit the histogram for $Q_1.$

set.seed(2019)
m = 10^6;  n = 5;  mu = 50;  sg = 7
x = rnorm(m*n, mu, sg)
MAT = matrix(x, nrow=m)      # each row a sample of             n
v1 = apply(MAT, 1, var)      # uses sample mean
v2 = rowSums((MAT - mu)^2)/n # uses population mean
q1 = (n-1)*v1 /sg^2
q2 = n*v2 / sg^2 

mean(q1);  var(q1)
[1] 3.997226   # aprx E(Q1) = 4
[1] 8.00637    # aprx Var(Q1) = 8
mean(q2);  var(q2)
[1] 4.997005   # aprx E(Q2) = 5
[1] 9.98925    # aprx Var(Q2) = 10

par(mfrow=c(1,2))  # enables 2 panels per plot
hist(q1, prob=T, br=50, col="skyblue2", ylim=c(0,.2))
 curve(dchisq(x, n-1), add=T, lwd=2)
 curve(dchisq(x, n), add=T, col="red", lwd=2, lty="dotted")
hist(q2, prob=T, br=50, col="skyblue2")
  curve(dchisq(x, n), add=T, lwd=2)       
par(mfrow=c(1,1)) 

Answer 2

Your understanding of degrees of freedom is correct for one of the two senses of the term. The vector $\left(X_1-\overline X,\ldots,X_n-\overline X\right)$ has $n-1$ degrees of freedom because it is subject to the constraint that the sum of the components must be $0,$ so if you know $n-1$ of them plus that constraint, then you know all of them.

The other sense of the term degrees of freedom is used when one speaks of a chi-square distribution with a specified number of degrees of freedom. Consider any orthonormal basis of the $(n-1)$-dimensional space of $n$-tuples in which the sum of the components is $0.$ Let $U_1,\ldots,U_{n-1}$ be the scalar components of $\left( X_1-\overline X, \ldots, X_n-\overline X \right)$ with respect to that basis. Then \begin{align} & \left( X_1-\overline X\right)^2 + \cdots + \left( X_n - \overline X\right)^2 = U_1^2 + \cdots + U_{n-1}^2 \\[5pt] & \text{and } U_1,\ldots,U_{n-1} \sim \text{i.i.d. } N(0, \sigma^2). \end{align}

Answer 3

Your understanding of degrees of freedom is correct. The difference essentially boils down to a subtle difference between $\mu$ and $\bar{X}$.

The sample mean $\bar{X}$ is determined by the values of the observed samples. As a result, there's some redundancy between $\bar{X}$ and the individual values of $x_i$. Suppose we knew $\bar{X}$ and the first $N-1$ values. That's enough, because we can write out the equation for $\bar{X}$ as $$\bar{X}=\frac{1}{N}x_1 +\frac{1}{N}x_2 + \frac{1}{N}x_3+ \ldots + \frac{1}{N}x_{N-1} + \frac{1}{N}x_N$$

A little bit of rearrangement lets us find that missing value: $$x_N = \bar{X} - \frac{N-1}{N}(x_1+x_2+\ldots +x_{N-1}) $$

This isn't the case when the population mean ($\mu$) is used, since it is not dependent on the observed data; it's known (or assumed to be known) ahead of time. You therefore need to use all $N$ values to produce calculate your test statistic.