2
$\begingroup$

According to Wiki: In statistics, the number of degrees of freedom is the number of values in the final calculation of a statistic that are free to vary.[1] Estimates of statistical parameters can be based upon different amounts of information or data. The number of independent pieces of information that go into the estimate of a parameter is called the degrees of freedom. In general, the degrees of freedom of an estimate of a parameter are equal to the number of independent scores that go into the estimate minus the number of parameters used as intermediate steps in the estimation of the parameter itself. For example, if the variance is to be estimated from a random sample of independent scores, then the degrees of freedom is equal to the number of independent scores (N) minus the number of parameters estimated as intermediate steps (one, namely, the sample mean) and is therefore equal to N-1.[2]

Reference:https://en.wikipedia.org/wiki/Degrees_of_freedom_(statistics)

I am confused that because we know the sample mean which means that $n$ observations are already known so why we can let $n-1$ observations be free to vary? I checked some videos and blogs: most of them explain that by using a example that if we have a sample of 3 and know the sample mean two of them are free to vary and the third observation will be determined by sample mean. But if we already know the sample mean why two of them are free to vary? I think these three observations are fixed.

$\endgroup$
6
  • 5
    $\begingroup$ Your last statement isn't necessarily true: consider three points with a sample mean of 5. The sets of observations (5,5,5) (0,5,10) (5,6,7) could all have given rise to this mean - and in fact infinitely many more sets could have. However, if I tell you the mean is 5 and the first two observations are (1,3), then the only option that makes those constraints fit is for the third to be 11. That's why you only have 2 'degrees of freedom'. $\endgroup$
    – PBulls
    Commented Dec 1, 2023 at 8:09
  • $\begingroup$ @PBulls Hi! I may further clarify my statement. I just mean that becuase if we know the sample mean which reveals that the prior condition is that we know each value of the observation in that sample because we can calculate the sample mean only with knowledge of these values. So why n-1 observations are free to vary? I think all observations in sample are fixed. $\endgroup$
    – Steven Xu
    Commented Dec 1, 2023 at 8:37
  • $\begingroup$ A large element of statistics is to draw inference beyond the data, e.g. how likely would a certain parameter value (like a mean) have been to produce the observed result. In this context the hypothetical observations that could have produced that mean aren't fixed. $\endgroup$
    – PBulls
    Commented Dec 1, 2023 at 8:40
  • 1
    $\begingroup$ A sample is a fixed realisation from a distribution that allows some variation. Every sample will be different and each experiment you have a different residual. The degrees of freedom describes the dimension of the distribution of the residuals. $\endgroup$ Commented Dec 1, 2023 at 8:44
  • 1
    $\begingroup$ A comment on English usage. I'd say that the term "degrees of freedom" can be awkward to use elegantly. In particular cases, it is relatively easy: we have 42 or 7 or 0 degrees of freedom, or on occasion only 1 degree of freedom. That's standard English grammar, bizarre though it may seem to learners. In the general case, I suggest "the number of degrees of freedom is" as correct if possibly seeming long-winded or pedantic. $\endgroup$
    – Nick Cox
    Commented Dec 1, 2023 at 9:22

2 Answers 2

7
$\begingroup$

According to Wiki: In statistics, the number of degrees of freedom is the number of values in the final calculation of a statistic that are free to vary.

I agree with you that this description is not very clear. Indeed, all $n$ of the variables are able to vary. The point is that they are constrained to a space that can be described by $n-p$ parameters.

Useful related questions

The following two images might be helpful:

Both images display the results of a low dimensional situation, with 3 variables such that they can plotted easily, but you may imagine how the logic is general for a higher number of variables.

illustration 1

illustration for a small sample size

Samples with $n=3$ observations $y_1$, $y_2$ and $y_3$ are fitted with a linear model of $p=2$ parameters.

The values $\hat{y}_i$ that the linear model can fit can be considered as a linear plane, the span of the vectors $x_1=[1,1,1]$ and $x_2=[1,2,3]$. The residual is the difference between the observation and the fitted point in the plane and this is always pointing in the same direction. That is, the residuals are a multiple of the vector $[1,-2,1]$ and can be described by a single value, the multiplication factor.

Illustration 2

example of distribution constraint to surface

This is an image from an early version of an answer here. The image relates to a chi-squared test for a table with 3 cells/values $x_A$, $x_B$ and $x_C$. Due to a constraint on their sum, they are fixed in a plane which you could see as a 2d distribution instead of a 3d distribution.


I just mean that becuase if we know the sample mean which reveals that the prior condition is that we know each value of the observation in that sample because we can calculate the sample mean only with knowledge of these values.

A sample is a fixed realisation from a distribution that allows some variation. Every sample will be different and each experiment you have a different residual. So for you specific observation the residual is fixed, but if you consider all potential outcomes then the residual is considered a random variable.

The number of degrees of freedom describes the dimension of the distribution of the residuals.

For example in the first image, points for different experiments/observations are plotted. In each case, the residual is always the value $a \cdot [1,-2,1]$, where $a$ can variate. It is a 1 dimensional space and 1 parameter can describe the distribution of the residuals.

$\endgroup$
4
$\begingroup$

I think PBulls and Sextus Empiricus gave a very clear, statistical answer already.

Maybe let me give 1 other, basic, non-statistical example:

Imagine you are a football manager and you have to put 11 players in a team where you have 11 fixed spots (1 for each player). So here: n = 11. Now you can put 10 players wherever you like (so you have 10 = 11-1 degrees of freedom), but the final player will be placed in the final open spot.

So back to your problem: The same happens with your sample: if you have a fixed mean (let's say $\mu$) from a sample of n = 4 observations, you can just randomly pick 3 number (3 = 4-1 degrees of freedom thus), but the final number should be in that way that your mean is still equal to the prespecified $\mu$ and therefore you are not free to choose this last number.

Now, in statistics, when you want to calculate some measure, like the mean, the degrees of freedom will tell you how many 'independent observations' you have that contribute to this estimate. In the example that you have 4 observations in total, you have 3 pieces of independent observations and you have 1 observation that is not free to vary anymore, but that is restricted by your other observations. So in that sense, you can say it's a way to quantify how many 'independent' data you have that is free to vary when calculating a statistic like the mean. Now it is hopefully clear that you of course first have your data and then calculate the mean, but this example shows how many of your observations are free to vary, when calculating this mean.

Closing example with numbers: let's say that you have a mean equal to 5 and a sample consisting of 4 observations (so n = 4). You can now ask the audience to shout 3 random numbers (these are you degrees of freedom, you can vary them as you like), and let's say they come up with 4, 5 and 10. Now the last number in your sample (let's call it X) is not free anymore, but should be such that $\frac{4 + 5 + 10 + X}{4} = 5$ (this is how you calculate the mean) and transforming this equation, you can find your last number: 1, which is now fixed, depending on your other 3 choosing values and the mean you want.

$\endgroup$
4
  • $\begingroup$ I am just wondering that why we can fixed the mean while varying the observations. From my understanding, we need to fixed the observations then we can know the sample mean. Why we can fixed the mean then let observations vary. This is my confusion. Thanks! $\endgroup$
    – Steven Xu
    Commented Dec 1, 2023 at 8:40
  • $\begingroup$ I updated my answer! $\endgroup$ Commented Dec 1, 2023 at 8:49
  • $\begingroup$ Hi Mathemagician777! Thanks for your detailed explanation. I think I am almost getting your point. To make calrify of my understanding, we first need to use data to calculate the mean and then we use the mean as a prior condition, so the degree of freedom is that the independent observations we need to get this prior condition. Is my understanding correct? $\endgroup$
    – Steven Xu
    Commented Dec 1, 2023 at 9:11
  • $\begingroup$ Yes, you first need your sample to calculate your mean and the degrees of freedom tell you how much of your sample is independent information. The same is done when you do a simulation to show that the beta-coefficients in a regression model are unbiased, you start from a population you know to prove your point, but in reality you don't know that and the same works here: we explain DF using that we 'know' the mean, whereas we actually just want to estimate it, just to prove here that DF denotes the number of observations that are free to vary. $\endgroup$ Commented Dec 1, 2023 at 9:22

Not the answer you're looking for? Browse other questions tagged or ask your own question.