Why is there a change in the number of degrees of freedom when the following modification is made?

Question

In the notes that I'm working through it says the following: "Let $X_1,...,X_n$ be a random sample from $N(\mu,\sigma)$

$$\sum^{n}_{i=1}\Bigg[\frac{(X_i-\mu)}{\sigma}\Bigg]^2$$ has a $\chi^2$ distribution with $n$ degrees of freedom.

Now if we modify this by replacing $\mu$ with $\overline{X}$ the distribution changes and we obtain: $$\sum^{n}_{i=1}\Bigg[\frac{(X_i-\overline{X})}{\sigma}\Bigg]^2$$ has a $\chi^2$ distribution with $n-1$ degrees of freedom."

My question is: why does the number of degrees of freedom change?

My understanding of what a degree of freedom is that the number of degrees of freedom is the number of values in the final calculation of a statistic that are free to vary. So surely as there are $n$ $X_i$ even when we introduce $\overline{X}$ the number of values that are free to change in the calculation of the statistic is still the same??

Answer 1

Suppose we have a random sample from $\mathsf{Norm}(\mu, \sigma).$

Let the $V_1 = S^2 = \frac{1}{n-1}\sum_{i=1}^n (X_i - \bar X)^2$ be the estimate of the population variance $\sigma^2$ when $\mu$ is unknown and estimated by $\bar X.$ Then $Q_1 = \frac{(n-1)V_1}{\sigma^2} \sim \mathsf{Chisq}(n-1).$

Let the $V_2 = \frac{1}{n}\sum_{i=1}^n (X_i - \mu)^2$ be the estimate of the population variance $\sigma^2$ when $\mu$ is known. Then $Q_2 = \frac{nV_2}{\sigma^2} \sim \mathsf{Chisq}(n).$

In the R code below, we choose $m = 10^6$ samples of size $n = 5$ from $\mathsf{Norm}(\mu = 50, \sigma = 7).$ Then we make histograms of $Q_1$ and $Q_2.$ Chi-squared densities with degrees of freedom $4$ and $5,$ respectively, fit the histograms. The density curve for $\mathsf{Chisq}(5)$ does not fit the histogram for $Q_1.$

set.seed(2019)
m = 10^6;  n = 5;  mu = 50;  sg = 7
x = rnorm(m*n, mu, sg)
MAT = matrix(x, nrow=m)      # each row a sample of             n
v1 = apply(MAT, 1, var)      # uses sample mean
v2 = rowSums((MAT - mu)^2)/n # uses population mean
q1 = (n-1)*v1 /sg^2
q2 = n*v2 / sg^2 

mean(q1);  var(q1)
[1] 3.997226   # aprx E(Q1) = 4
[1] 8.00637    # aprx Var(Q1) = 8
mean(q2);  var(q2)
[1] 4.997005   # aprx E(Q2) = 5
[1] 9.98925    # aprx Var(Q2) = 10

par(mfrow=c(1,2))  # enables 2 panels per plot
hist(q1, prob=T, br=50, col="skyblue2", ylim=c(0,.2))
 curve(dchisq(x, n-1), add=T, lwd=2)
 curve(dchisq(x, n), add=T, col="red", lwd=2, lty="dotted")
hist(q2, prob=T, br=50, col="skyblue2")
  curve(dchisq(x, n), add=T, lwd=2)       
par(mfrow=c(1,1)) 

Answer 2

Your understanding of degrees of freedom is correct for one of the two senses of the term. The vector $\left(X_1-\overline X,\ldots,X_n-\overline X\right)$ has $n-1$ degrees of freedom because it is subject to the constraint that the sum of the components must be $0,$ so if you know $n-1$ of them plus that constraint, then you know all of them.

The other sense of the term degrees of freedom is used when one speaks of a chi-square distribution with a specified number of degrees of freedom. Consider any orthonormal basis of the $(n-1)$-dimensional space of $n$-tuples in which the sum of the components is $0.$ Let $U_1,\ldots,U_{n-1}$ be the scalar components of $\left( X_1-\overline X, \ldots, X_n-\overline X \right)$ with respect to that basis. Then \begin{align} & \left( X_1-\overline X\right)^2 + \cdots + \left( X_n - \overline X\right)^2 = U_1^2 + \cdots + U_{n-1}^2 \\[5pt] & \text{and } U_1,\ldots,U_{n-1} \sim \text{i.i.d. } N(0, \sigma^2). \end{align}

Answer 3

Your understanding of degrees of freedom is correct. The difference essentially boils down to a subtle difference between $\mu$ and $\bar{X}$.

The sample mean $\bar{X}$ is determined by the values of the observed samples. As a result, there's some redundancy between $\bar{X}$ and the individual values of $x_i$. Suppose we knew $\bar{X}$ and the first $N-1$ values. That's enough, because we can write out the equation for $\bar{X}$ as $$\bar{X}=\frac{1}{N}x_1 +\frac{1}{N}x_2 + \frac{1}{N}x_3+ \ldots + \frac{1}{N}x_{N-1} + \frac{1}{N}x_N$$

A little bit of rearrangement lets us find that missing value: $$x_N = \bar{X} - \frac{N-1}{N}(x_1+x_2+\ldots +x_{N-1}) $$

This isn't the case when the population mean ($\mu$) is used, since it is not dependent on the observed data; it's known (or assumed to be known) ahead of time. You therefore need to use all $N$ values to produce calculate your test statistic.

Answer 4

The case of two observations is very illustrative. In the example below the points are distributed as:

$$X_i \sim N(3,1)$$

The estimated mean is a projection onto the line $x_1 = x_2$. Now you can see that the distance from the observation to the mean can be partitioned into two perpendicular components. The distance of the observation to the estimated mean is one component less. This is the cause for the degree of freedom change.

In the same image, you can see that the distance $2$ terms $x_1-\bar{x}$ and $x_2-\bar{x}$ depend on only one parameter (the direction is fixed)