Does the determination of the mean and SD imply the loss of one or two degrees of freedom?

Question

I'm facing some doubts in understanding how degrees of freedom are considered in distributions.

In particular let's refer to $t$ Student variable, that is

$$t=\frac{x-\bar{x}}{\hat{s}}=\frac{x-\bar{x}}{\sqrt{\frac{\sum(x_i-\bar{x})^2}{N-1}}}\tag{1}$$

Where $x$ is a gaussian variable, $\bar{x}$ is the mean value, $\hat{s}=\sqrt{\frac{\sum(x_i-\bar{x})^2}{N-1}}$ is the standard deviation taken from data.

Student probability density function is $$f(t)=C (1+\frac{t^2}{\nu})^{-\frac{\nu+1}{2}}\tag{2}$$

And on my textbook I find $\nu=N-1$ "because in $(1)$ appears the mean value $\bar{x}$, calculated from data, which implies the loss of a degree of freedom".

Question: Shouldn't it be $\nu=N-2$? In $(1)$ I have both $\hat{s}$ and $\bar{x}$ so there are two parameters determined from data.

On the other hand in the second form I wrote in $(1)$, $\hat{s}$ does not appear, so maybe only $\bar{x}$ should be considered as a constraint on data. But this does not make a lot of sense.

So in these cases where both the mean value and the standard deviation are determined from data, are the degrees of freedom lost 2 or only 1?

This is kind of a more general doubt: when more than one parameter is determined from data, but in some ways these parameters are related (as it is for $\bar{x}$ and $\hat{s}$), how many degrees of freedom are lost if all these parameter are considered?

Say for instance I determine $q$ parameters $p_1,p_2,...,p_q$ from the same set of data. All the parameters $p_2,...,p_q$ can be expressed as functions of data and $p_1$. Now I consider all the parameters together: how many degrees of freedom did I lose? $q$ or just $1$?

Answer 1

The T-distribution is defined as the distribution of the ratio of a standard normal random variable and an independent scaled-chi random variable. Its degrees-of-freedom-parameter is equal to the degrees-of-freedom parameter for the chi random variable in its denominator. So the DF parameter is a matter of determining the degrees-of-freedom of the variance estimator you are using.

Remember: The T-distribution only arises when you take the ratio of a normal random variable and a denominator which is some kind of standard deviation estimator (square root of a variance estimator). This presumes that there is already a variance estimator in the picture. The loss of degrees-of-freedom then occurs from the mean estimate (or in the context of regression, from multiple coefficient estimates).

---

It is possible to form quantities similar to the one you have shown, and find their distributions. Suppose we have $X_1, ..., X_n \sim \text{IID N}(\mu, \sigma^2)$ and we form some standardised value. If we assume that $\mu$ is known but $\sigma$ is unknown, we would standardise by defining the T-statistic:

$$T_\mu \equiv \frac{X_i - \mu}{S_\mu} = \frac{X_i - \mu}{\sigma} / \frac{S_\mu}{\sigma} \sim \text{T} (n),$$

where $S_\mu^2 \equiv \frac{1}{n} \sum_{i=1}^n (X_i - \mu)^2$ is the sample variance estimator with known $\mu$. The quantity $S_\mu / \sigma$ is a scaled-chi random variable with $n$ degrees-of-freedom, so the statistic $T_\mu$ has a T-distribution with $n$ degrees-of-freedom. This is a baseline case where there has not been any loss of degrees-of-freedom, even though we have estimated the variance.

Now, in the case where $\mu$ is also unknown we would replace the known mean $\mu$ in the variance estimator with the sample mean $\bar{x}$ we have:

$$T \equiv \frac{X_i - \mu}{S} = \frac{X_i - \mu}{\sigma} / \frac{S}{\sigma} \sim \text{T}(n-1),$$

where $S^2 \equiv \frac{1}{n-1} \sum_{i=1}^n (X_i - \bar{x})^2$ is the sample variance estimator with unknown $\mu$. The quantity $S / \sigma$ is a scaled-chi random variable with $n-1$ degrees-of-freedom, so the statistic $T$ has a T-distribution with $n-1$ degrees-of-freedom. We have lost one degree-of-freedom due to estimating the mean inside the variance estimator.

---

Hopefully this assists you in understanding this issue. The concept of degrees-of-freedom, within the context of talking about the T-distribution, presumes that there is already some variance estimator being used for the studentisation. Estimating the mean parameter (or coefficient parameters in a regression) alters this variance estimator by making it less variable, and this entails a loss of degrees-of-freedom.

Answer 2

Let's consider an example to understand degrees of freedom:

Pretend we have 5 observations, $(1, 2, 1, 3, 5)$. If I tell you the mean of this data set ($2.4$) but not the values of the observations themselves, you can make up four values without changing the mean. If you pick $(3, 4, 3, 5)$ as your first four observations, then the last number to choose must be $-3$ if the mean is fixed at $2.4$. If we only care about the mean, then we have one equation and one unknown.

If you have $n$ observations with a fixed mean, you have the freedom to pick any $n - 1$ numbers you want without changing the mean -- but the $n^{th}$ observation is determined. Notice however, I chose the value of $2.4$ in the paragraph above arbitrarily, so I could have chosen something else. Therefore, I have $n - 1$ degrees of freedom from the data and $1$ degree of freedom because I picked the mean, so I have $n$ degrees of freedom if I estimate 1 parameter.

Now, let's say I tell you the mean and the standard deviation: for the same sample of $(1, 2, 1, 3, 5)$, the mean is $2.4$ and the standard deviation is $1.673$. Now I can pick three of the five numbers, and the last two will be determined (two equations, two unknowns). The parameters are a little different however, because the sample standard deviation is a function of the sample mean -- they are not independent of each other. This means that I have $n - 2$ degrees of freedom from the data, but still only $1$ degree of freedom from the parameters, for a total of $n - 1$ degrees of freedom.

See this Stack Exchange question for more information.