7
$\begingroup$

I'm facing some doubts in understanding how degrees of freedom are considered in distributions.

In particular let's refer to $t$ Student variable, that is

$$t=\frac{x-\bar{x}}{\hat{s}}=\frac{x-\bar{x}}{\sqrt{\frac{\sum(x_i-\bar{x})^2}{N-1}}}\tag{1}$$

Where $x$ is a gaussian variable, $\bar{x}$ is the mean value, $\hat{s}=\sqrt{\frac{\sum(x_i-\bar{x})^2}{N-1}}$ is the standard deviation taken from data.

Student probability density function is $$f(t)=C (1+\frac{t^2}{\nu})^{-\frac{\nu+1}{2}}\tag{2}$$

And on my textbook I find $\nu=N-1$ "because in $(1)$ appears the mean value $\bar{x}$, calculated from data, which implies the loss of a degree of freedom".

Question: Shouldn't it be $\nu=N-2$? In $(1)$ I have both $\hat{s}$ and $\bar{x}$ so there are two parameters determined from data.

On the other hand in the second form I wrote in $(1)$, $\hat{s}$ does not appear, so maybe only $\bar{x}$ should be considered as a constraint on data. But this does not make a lot of sense.

So in these cases where both the mean value and the standard deviation are determined from data, are the degrees of freedom lost 2 or only 1?

This is kind of a more general doubt: when more than one parameter is determined from data, but in some ways these parameters are related (as it is for $\bar{x}$ and $\hat{s}$), how many degrees of freedom are lost if all these parameter are considered?

Say for instance I determine $q$ parameters $p_1,p_2,...,p_q$ from the same set of data. All the parameters $p_2,...,p_q$ can be expressed as functions of data and $p_1$. Now I consider all the parameters together: how many degrees of freedom did I lose? $q$ or just $1$?

$\endgroup$
  • $\begingroup$ When you estimated s^ you already lost one d.f., so maybe it's embedded within it, and when you use s^ you don't need to take it again into account? $\endgroup$ – EBH Aug 11 '16 at 20:55
  • $\begingroup$ You are correct: this does not make a lot of sense. That is why such an expression for $t$ is never used! In practice we compare the mean $\bar x$ of data $x_1, \ldots, x_n$ to some other statistic or a number, but we do not use $t$ to compare the individual $x_i$ to their mean. I am confident the expression in your textbook differs from what you are quoting here. $\endgroup$ – whuber May 7 '17 at 21:13
1
$\begingroup$

The T-distribution is defined as the distribution of the ratio of a standard normal random variable and an independent scaled-chi random variable. Its degrees-of-freedom-parameter is equal to the degrees-of-freedom parameter for the chi random variable in its denominator. So the DF parameter is a matter of determining the degrees-of-freedom of the variance estimator you are using.

Remember: The T-distribution only arises when you take the ratio of a normal random variable and a denominator which is some kind of standard deviation estimator (square root of a variance estimator). This presumes that there is already a variance estimator in the picture. The loss of degrees-of-freedom then occurs from the mean estimate (or in the context of regression, from multiple coefficient estimates).


It is possible to form quantities similar to the one you have shown, and find their distributions. Suppose we have $X_1, ..., X_n \sim \text{IID N}(\mu, \sigma^2)$ and we form some standardised value. If we assume that $\mu$ is known but $\sigma$ is unknown, we would standardise by defining the T-statistic:

$$T_\mu \equiv \frac{X_i - \mu}{S_\mu} = \frac{X_i - \mu}{\sigma} / \frac{S_\mu}{\sigma} \sim \text{T} (n),$$

where $S_\mu^2 \equiv \frac{1}{n} \sum_{i=1}^n (X_i - \mu)^2$ is the sample variance estimator with known $\mu$. The quantity $S_\mu / \sigma$ is a scaled-chi random variable with $n$ degrees-of-freedom, so the statistic $T_\mu$ has a T-distribution with $n$ degrees-of-freedom. This is a baseline case where there has not been any loss of degrees-of-freedom, even though we have estimated the variance.

Now, in the case where $\mu$ is also unknown we would replace the known mean $\mu$ in the variance estimator with the sample mean $\bar{x}$ we have:

$$T \equiv \frac{X_i - \mu}{S} = \frac{X_i - \mu}{\sigma} / \frac{S}{\sigma} \sim \text{T}(n-1),$$

where $S^2 \equiv \frac{1}{n-1} \sum_{i=1}^n (X_i - \bar{x})^2$ is the sample variance estimator with unknown $\mu$. The quantity $S / \sigma$ is a scaled-chi random variable with $n-1$ degrees-of-freedom, so the statistic $T$ has a T-distribution with $n-1$ degrees-of-freedom. We have lost one degree-of-freedom due to estimating the mean inside the variance estimator.


Hopefully this assists you in understanding this issue. The concept of degrees-of-freedom, within the context of talking about the T-distribution, presumes that there is already some variance estimator being used for the studentisation. Estimating the mean parameter (or coefficient parameters in a regression) alters this variance estimator by making it less variable, and this entails a loss of degrees-of-freedom.

$\endgroup$
0
$\begingroup$

Let's consider an example to understand degrees of freedom:

Pretend we have 5 observations, $(1, 2, 1, 3, 5)$. If I tell you the mean of this data set ($2.4$) but not the values of the observations themselves, you can make up four values without changing the mean. If you pick $(3, 4, 3, 5)$ as your first four observations, then the last number to choose must be $-3$ if the mean is fixed at $2.4$. If we only care about the mean, then we have one equation and one unknown.

If you have $n$ observations with a fixed mean, you have the freedom to pick any $n - 1$ numbers you want without changing the mean -- but the $n^{th}$ observation is determined. Notice however, I chose the value of $2.4$ in the paragraph above arbitrarily, so I could have chosen something else. Therefore, I have $n - 1$ degrees of freedom from the data and $1$ degree of freedom because I picked the mean, so I have $n$ degrees of freedom if I estimate 1 parameter.

Now, let's say I tell you the mean and the standard deviation: for the same sample of $(1, 2, 1, 3, 5)$, the mean is $2.4$ and the standard deviation is $1.673$. Now I can pick three of the five numbers, and the last two will be determined (two equations, two unknowns). The parameters are a little different however, because the sample standard deviation is a function of the sample mean -- they are not independent of each other. This means that I have $n - 2$ degrees of freedom from the data, but still only $1$ degree of freedom from the parameters, for a total of $n - 1$ degrees of freedom.

See this Stack Exchange question for more information.

$\endgroup$
  • 1
    $\begingroup$ You were headed in a good direction, but the comments at the end are incorrect. The sample SD is not a function of the sample mean (except for samples of 1). The remarks about parameters seem to come out of nowhere, implicitly confuse two concepts of "independent" (statistical and functional), and don't have any clear bearing on the question about the sampling distributions of statistics. The answers to the question on the math site are restricted and unimaginative. The truth is far more complex and interesting: see our thread on this subject at stats.stackexchange.com/questions/16921. $\endgroup$ – whuber Dec 7 '17 at 23:20
  • 1
    $\begingroup$ I usually use this example to teach my freshman students, but it obviously fails under any checks of rigor. I thought it would be appropriate given the level of the question, but it appears I was wrong. I don't know how to discuss degrees of freedom technically with out talking about the rank of the Hat matrix. Thank you for the link and feedback. I'll check it out. $\endgroup$ – Gabriel J. Odom Dec 7 '17 at 23:28
  • $\begingroup$ @whuber, thank you for that thread. I have a PhD in stats, and I didn't know half of what you mentioned. I feel like a complete moron now. $\endgroup$ – Gabriel J. Odom Dec 7 '17 at 23:44
  • $\begingroup$ No need to feel that way! The reason many of us hang out here is we often read posts that reveal how little we know (or even better, how what we thought we knew isn't so), because we learn so much from them. The bolder (or stupider) ones, like myself, learn even more by venturing frequently to answer and comment, where our mistakes become evident for all to see. (I made my last really stupid comment only five minutes ago... .) $\endgroup$ – whuber Dec 8 '17 at 0:31
  • $\begingroup$ Thanks for the encouragement Professor @whuber. I really appreciate it :) $\endgroup$ – Gabriel J. Odom Dec 8 '17 at 19:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.