In a $t$-distribution, what is the meaning of "degrees of freedom"?

Question

I know that the $t$-distribution has one parameter: the number of degrees of freedom (df).

I also know that $\mathrm{df} = n - 1$.

However, what EXACTLY does the $n$ represent?

I have heard several meanings:

1. "The number of data points in your dataset." This doesn't sound right. If I have $100$ data points, but the $t$-distribution with $\mathrm{df}=50$ models the distribution better than $\mathrm{df}=99$, then why should I force the $\mathrm{df}$ to be 99?

2. "The number of normal random variables that you use to define the $t$-statistic from the sample mean $\bar{x}$." This doesn't sound right, either. I know that

   $$t = \frac{\bar{x} - \mu}{s / \sqrt{n}}.$$

   However, that looks more like a mathematical relationship rather than a definition of $t$. What if I don't start with the normal random variables?

Answer 1

Let's discuss some distinct notions.

1. The t-distribution doesn't have an $n$. At all. Anywhere.

   It has a degrees-of-freedom parameter, conventionally that's $\nu$.

   It's simply a probability distribution (or rather, a family of them). Here's the density of a (standard) t at a variety of $\nu$ values:

   

   Original wikimedia image by Skbkekas Feb 2010 enhanced by Ika July 2013 used under CC-BY-3.0 link

   Now one way to obtain such a distribution is to divide a standard normal random variable by the square root of (an independent-of-the-numerator chi-squared divided by its degrees of freedom), i.e. $T=\frac{Z}{\sqrt{{Q}/{\nu}}}$ for $Q\sim \chi^2_\nu$ independent of $Z$. The chi-squared with $\nu$ d.f. can - if $\nu$ is a positive integer - in turn be obtained as a sum squares of independent standard normals. But the t-distribution exists as a mathematical object itself, it doesn't have to arise via this mechanism.

   Nor does it need to have its degrees of freedom parameter be an integer.

2. In a hypothesis testing situation, in particular a one-sample test or a paired test of means, under a set of assumptions, you can derive that a specific test statistic will have a t-distribution when $H_0$ is true.

   In that specific situation, the degrees-of-freedom parameter of the t-distribution comes out to be $n-1$, one less than the number of observations in the sample, $n$ (or one less than the number of observed pair-differences in a paired test). Here the degrees of freedom arise from the estimate of the variance that's used in the t-statistic. [If you did some other t-test, you might get that the d.f. are not $1$ lower than $n$. e.g. in a test of slope in a simple regression under the usual regression assumptions, it's still a t-test but the d.f. would be $n-2$.]

   You get a single value for your test statistic that would be distributed as a $t$ if $H_0$ were true (and if the assumptions held in that circumstance).

   In this example, the data set, y had 23 observations in it.

   > t.test(y,mu=100)
   
            One Sample t-test
   
    data:  y
    t = 2.0833, df = 22, p-value = 0.04906
    alternative hypothesis: true mean is not equal to 100
    95 percent confidence interval:
     100.0343 115.1565
    sample estimates:
    mean of x 
     107.5954 
   

   We get a single test statistic, 2.0833.

   If H0 were true, this would be a random value drawn from a standard t distribution with 22 d.f.

   As it turns out (though we can't know this from the data) the sample was not drawn from the hypothesized distribution, but was simulated from one with a slightly larger mean (one shifted right by $\frac{2}{15}\approx 0.133$ standard deviations).

   

   In this case the test statistic has a different distribution, one that tends to produce larger values for the test statistic than you'd see if $H_0$ was true.

   In this testing circumstance, the t-distribution is not a model for the observations, only for the statistic (and then only given the two sets of mentioned conditions).

   If the data had a $t$-distribution with finite $\nu$, the statistic could not be $t$-distributed; the $t$-statistic's $\nu$ parameter would then be irrelevant, as there isn't a $t$ distribution for the statistic at all.

3. The above use in a test is entirely distinct from using a $t$-distribution itself as a model for data.

   In the case of using it as a model for data, the degrees of freedom parameter has nothing to do with the sample size. You might have $n=1000$ observations and $\nu=2$ or you might have $n=3$ and $\nu=75$. They're unconnected.

   Here's a histogram of 3000 values from a t-distribution with $\nu=4.2$ (as mentioned above, the d.f. don't have to be integer in the t-distribution, and often aren't when modelling data). The density they were drawn from is also shown.

   

   Again, to emphasize the distinction, if I believed I had data drawn from a t-distribution with low d.f., I would not use a t-test as a one-sample location test for that case; even if we fixed the issue with nominal significance levels being wrong (because the test statistic wouldn't have a t-distribution), it would still have low power relative to a better choice of test.

Answer 2

Sometime, it helps to go back to the origin, where a new concept or distribution or terminology was introcuced. This is because, as others use these new concepts, and adapt/adjust their definitions, and the teaching of these new concepts evolve in various ways, things which were initially simple, precise, and clear, can become muddled.
The seminal paper is that of Gosset (aka Student), "The probable error of a mean". In it, Gosset defines the t distribution as the "frequency distribution of the means of samples drawn from a normal population, when these values are measured from the mean of the population in terms of the standard deviation of the sample". So, if you have iid normally distributed samples of size $n$, then the sampling distribution of the variable $z=\dfrac {(\bar x - \mu)} {s}$ will follow a t distribution of $df=n-1$.
So it is not the original sample which the t distribution is supposed to "fit", but the sampling distribution of this variable $z$ (which is a "standardized" -i.e. mean=0, sd<1- version of the sampling distribution of the means of such samples). The original sample is supposed to folllow a normal distribution (that is what the t distribution assumes). So trying to fit a t to the sample is not applicable here.
And trying to fit a t with a different df to the sampling distribution of the means is also not aplicable; the t with $df=n-1$ is the one which best fits the means.
And it is because it best fits the standardized distribution of the means (and better than the z-distribution) that it is used in a t test (which is better than a z test for testing means).
I am not exactly sure where the 2nd "definition" comes from (Googled it: nothing except the OP's questions). As written, it does not make sense.
Now, if you are simply trying to model a random sample of size $n$ with a t-distribution of $df=m \ne n$, you can absolutely do it. But if you want to estimate the mean of a (close to) normally distributed sample, you have to use a t-distribution with $df=n-1$.
Last, just to thicken the plot, I shall simply mention the 2-sample Welch t-test, where now the df becomes fractional, as described here or here on CV. If anyone can find an intuitive interpretation thereof, I would be immensively curious about it...

Answer 3

Here's the rigorous definition of the t-distribution:

Let $X_1$ and $X_2$ be independent random variables, $X_1\sim N(0,1)$, $X_2\sim \chi^2(n)$, then the distribution of $$ t=\frac{X_1}{\sqrt{X_2/n}} $$ is called a t-distribution with $n$ degrees of freedom (denoted as $t(n)$). The $n$ is just a parameter of the distribution.

However, if we look at the definition of a $\chi^2$-distribution:

Let $X_1,\cdots,X_n$ be i.i.d. random variables, $X_i\sim N(0,1)$, then the distribution of $$ \chi^2=\sum_{i=1}^nX_i^2 $$ is called a $\chi^2$-distribution with degrees of freedom $n$ (denoted as $\chi^2(n)$). In this case, the "degrees of freedom" is just how many terms you are adding up together. And because the t-distribution is defined using a $\chi^2$-distribution, it inherits a "degrees of freedom" parameter from the $\chi^2$-distribution.

And in the case of the F-distribution, there is not one, but two "degrees of freedom" parameters: $F(m,n)$, because the definition of the F-distribution requires two $\chi^2$-distributions.

---

(By the way, the statement (2) is clearly a theorem, not a definition; you have to prove that the t-statistic follows a t-distribution of $n-1$ d.f. when the sample size is $n$, and the population follows a normal distribution.)

Answer 4

For a t-distribution, df = n-1. What does n represent?

$n$ is simply the sample size.

The t-distribution with $\nu = n-1$ degrees of freedom occurs as the sample distribution of the t-statistic for a normal distributed sample with sample size $n$.

But, seemingly there is more going on with you question and you consider cases where there is no normal distributed sample.

---

What if I don't start with the normal random variables.

Then you won't have (exactly) a t-distribution for the t-statistic. You might have something approximately similar. It depends on the discrepancies. But with small discrepancies the model might still perform well (Which statistical analysis should I perform if the data sets are not normally distributed?).

1. "The number of data points in your dataset." This doesn't sound right. If I have $100$ data points, but the $t$-distribution with $\mathrm{df}=50$ models the distribution better than $\mathrm{df}=99$, then why should I force the $\mathrm{df}$ to be 99?

The use of $\nu = n-1$ is based on the assumption of normal distributed data.

If

but the $t$-distribution with $\mathrm{df}=50$ models the distribution better

, then use that different t-distribution

^{(Not a general advise, because I don't understand how you assessed that $\mathrm{df}=50$ is modeling better. But yes, sometimes a distribution with a different number of degrees of freedom could be better)}

But it won't be the situation of normal distributed data. That assumption is where the $\nu = n-1$ comes from, and indeed it can be different when you don't make that assumption.

A similar example is described here for adjusting the degrees of freedom in an F-distribution: F-test for equality of variance for truncated normal distributions

Answer 5

$N$ is the sample size, not the "number of data points in your dataset". Remember that the standard error to the sample mean is given by: $\sigma_{\overline{x}} = \frac{\sigma}{\sqrt{N}}$. But wait a minute, how am I supposed to compute this when I don't even know $\sigma$? Well, you were told to just use the sample standard deviation $s$ because it probably approximates the population standard deviation and thus allows you to use the normal distribution to solve all your problems. Does $s$ approximate $\sigma$ well? Well, not always.

When you do this for small enough sample sizes (small N), it's actually a poor approximation to the normal distribution and you should actually use the t-distribution instead. But let's say for $N<30$, every value of $N$ you go down by, the more variability there will be. Thus, there will be a different distribution for every value of $N$ you pick. Hopefully this establishes the connection on why $N$ is so important for your t-distributions and why it is the only parameter that matters.

So why $N-1$ instead of $N$? It has to do with the fact that you want one degree of freedom to approximate the mean (which is obviously important for solving problems), so that leaves $N-1$ to approximate the standard deviation of your small sample.

NB: 30 was an arbitrary number that I picked because that's what you see in a lot of books. In principle, there will be error with $N=40,50,60$, really any value of $N$. But it's usually most sensitive for ones less than 100, and you get a pretty close approximation to the normal distribution for $N>30$, close enough for a lot of applications.