# How to understand degrees of freedom?

From Wikipedia, there are three interpretations of the degrees of freedom of a statistic:

In statistics, the number of degrees of freedom is the number of values in the final calculation of a statistic that are free to vary.

Estimates of statistical parameters can be based upon different amounts of information or data. The number of independent pieces of information that go into the estimate of a parameter is called the degrees of freedom (df). In general, the degrees of freedom of an estimate of a parameter is equal to the number of independent scores that go into the estimate minus the number of parameters used as intermediate steps in the estimation of the parameter itself (which, in sample variance, is one, since the sample mean is the only intermediate step).

Mathematically, degrees of freedom is the dimension of the domain of a random vector, or essentially the number of 'free' components: how many components need to be known before the vector is fully determined.

The bold words are what I don't quite understand. If possible, some mathematical formulations will help clarify the concept.

Also do the three interpretations agree with each other?

• Check out this explanation Jul 28 '10 at 10:26
• Also see this question "What are degrees of freedom?" Oct 12 '11 at 22:12
• Jul 26 at 23:01
• See Ye 1998 for a generalization of degrees of freedom. Oct 7 at 0:38

This is a subtle question. It takes a thoughtful person not to understand those quotations! Although they are suggestive, it turns out that none of them is exactly or generally correct. I haven't the time (and there isn't the space here) to give a full exposition, but I would like to share one approach and an insight that it suggests.

Where does the concept of degrees of freedom (DF) arise? The contexts in which it's found in elementary treatments are:

• The Student t-test and its variants such as the Welch or Satterthwaite solutions to the Behrens-Fisher problem (where two populations have different variances).

• The Chi-squared distribution (defined as a sum of squares of independent standard Normals), which is implicated in the sampling distribution of the variance.

• The F-test (of ratios of estimated variances).

• The Chi-squared test, comprising its uses in (a) testing for independence in contingency tables and (b) testing for goodness of fit of distributional estimates.

In spirit, these tests run a gamut from being exact (the Student t-test and F-test for Normal variates) to being good approximations (the Student t-test and the Welch/Satterthwaite tests for not-too-badly-skewed data) to being based on asymptotic approximations (the Chi-squared test). An interesting aspect of some of these is the appearance of non-integral "degrees of freedom" (the Welch/Satterthwaite tests and, as we will see, the Chi-squared test). This is of especial interest because it is the first hint that DF is not any of the things claimed of it.

We can dispose right away of some of the claims in the question. Because "final calculation of a statistic" is not well-defined (it apparently depends on what algorithm one uses for the calculation), it can be no more than a vague suggestion and is worth no further criticism. Similarly, neither "number of independent scores that go into the estimate" nor "the number of parameters used as intermediate steps" are well-defined.

"Independent pieces of information that go into [an] estimate" is difficult to deal with, because there are two different but intimately related senses of "independent" that can be relevant here. One is independence of random variables; the other is functional independence. As an example of the latter, suppose we collect morphometric measurements of subjects--say, for simplicity, the three side lengths $X$, $Y$, $Z$, surface areas $S=2(XY+YZ+ZX)$, and volumes $V=XYZ$ of a set of wooden blocks. The three side lengths can be considered independent random variables, but all five variables are dependent RVs. The five are also functionally dependent because the codomain (not the "domain"!) of the vector-valued random variable $(X,Y,Z,S,V)$ traces out a three-dimensional manifold in $\mathbb{R}^5$. (Thus, locally at any point $\omega\in\mathbb{R}^5$, there are two functions $f_\omega$ and $g_\omega$ for which $f_\omega(X(\psi),\ldots,V(\psi))=0$ and $g_\omega(X(\psi),\ldots,V(\psi))=0$ for points $\psi$ "near" $\omega$ and the derivatives of $f$ and $g$ evaluated at $\omega$ are linearly independent.) However--here's the kicker--for many probability measures on the blocks, subsets of the variables such as $(X,S,V)$ are dependent as random variables but functionally independent.

Having been alerted by these potential ambiguities, let's hold up the Chi-squared goodness of fit test for examination, because (a) it's simple, (b) it's one of the common situations where people really do need to know about DF to get the p-value right and (c) it's often used incorrectly. Here's a brief synopsis of the least controversial application of this test:

• You have a collection of data values $(x_1, \ldots, x_n)$, considered as a sample of a population.

• You have estimated some parameters $\theta_1, \ldots, \theta_p$ of a distribution. For example, you estimated the mean $\theta_1$ and standard deviation $\theta_2 = \theta_p$ of a Normal distribution, hypothesizing that the population is normally distributed but not knowing (in advance of obtaining the data) what $\theta_1$ or $\theta_2$ might be.

• In advance, you created a set of $k$ "bins" for the data. (It may be problematic when the bins are determined by the data, even though this is often done.) Using these bins, the data are reduced to the set of counts within each bin. Anticipating what the true values of $(\theta)$ might be, you have arranged it so (hopefully) each bin will receive approximately the same count. (Equal-probability binning assures the chi-squared distribution really is a good approximation to the true distribution of the chi-squared statistic about to be described.)

• You have a lot of data--enough to assure that almost all bins ought to have counts of 5 or greater. (This, we hope, will enable the sampling distribution of the $\chi^2$ statistic to be approximated adequately by some $\chi^2$ distribution.)

Using the parameter estimates, you can compute the expected count in each bin. The Chi-squared statistic is the sum of the ratios

$$\frac{(\text{observed}-\text{expected})^2}{\text{expected}}.$$

This, many authorities tell us, should have (to a very close approximation) a Chi-squared distribution. But there's a whole family of such distributions. They are differentiated by a parameter $\nu$ often referred to as the "degrees of freedom." The standard reasoning about how to determine $\nu$ goes like this

I have $k$ counts. That's $k$ pieces of data. But there are (functional) relationships among them. To start with, I know in advance that the sum of the counts must equal $n$. That's one relationship. I estimated two (or $p$, generally) parameters from the data. That's two (or $p$) additional relationships, giving $p+1$ total relationships. Presuming they (the parameters) are all (functionally) independent, that leaves only $k-p-1$ (functionally) independent "degrees of freedom": that's the value to use for $\nu$.

The problem with this reasoning (which is the sort of calculation the quotations in the question are hinting at) is that it's wrong except when some special additional conditions hold. Moreover, those conditions have nothing to do with independence (functional or statistical), with numbers of "components" of the data, with the numbers of parameters, nor with anything else referred to in the original question.

Let me show you with an example. (To make it as clear as possible, I'm using a small number of bins, but that's not essential.) Let's generate 20 independent and identically distributed (iid) standard Normal variates and estimate their mean and standard deviation with the usual formulas (mean = sum/count, etc.). To test goodness of fit, create four bins with cutpoints at the quartiles of a standard normal: -0.675, 0, +0.657, and use the bin counts to generate a Chi-squared statistic. Repeat as patience allows; I had time to do 10,000 repetitions.

The standard wisdom about DF says we have 4 bins and 1+2 = 3 constraints, implying the distribution of these 10,000 Chi-squared statistics should follow a Chi-squared distribution with 1 DF. Here's the histogram:

The dark blue line graphs the PDF of a $\chi^2(1)$ distribution--the one we thought would work--while the dark red line graphs that of a $\chi^2(2)$ distribution (which would be a good guess if someone were to tell you that $\nu=1$ is incorrect). Neither fits the data.

You might expect the problem to be due to the small size of the data sets ($n$=20) or perhaps the small size of the number of bins. However, the problem persists even with very large datasets and larger numbers of bins: it is not merely a failure to reach an asymptotic approximation.

Things went wrong because I violated two requirements of the Chi-squared test:

1. You must use the Maximum Likelihood estimate of the parameters. (This requirement can, in practice, be slightly violated.)

2. You must base that estimate on the counts, not on the actual data! (This is crucial.)

The red histogram depicts the chi-squared statistics for 10,000 separate iterations, following these requirements. Sure enough, it visibly follows the $\chi^2(1)$ curve (with an acceptable amount of sampling error), as we had originally hoped.

The point of this comparison--which I hope you have seen coming--is that the correct DF to use for computing the p-values depends on many things other than dimensions of manifolds, counts of functional relationships, or the geometry of Normal variates. There is a subtle, delicate interaction between certain functional dependencies, as found in mathematical relationships among quantities, and distributions of the data, their statistics, and the estimators formed from them. Accordingly, it cannot be the case that DF is adequately explainable in terms of the geometry of multivariate normal distributions, or in terms of functional independence, or as counts of parameters, or anything else of this nature.

We are led to see, then, that "degrees of freedom" is merely a heuristic that suggests what the sampling distribution of a (t, Chi-squared, or F) statistic ought to be, but it is not dispositive. Belief that it is dispositive leads to egregious errors. (For instance, the top hit on Google when searching "chi squared goodness of fit" is a Web page from an Ivy League university that gets most of this completely wrong! In particular, a simulation based on its instructions shows that the chi-squared value it recommends as having 7 DF actually has 9 DF.)

With this more nuanced understanding, it's worthwhile to re-read the Wikipedia article in question: in its details it gets things right, pointing out where the DF heuristic tends to work and where it is either an approximation or does not apply at all.

A good account of the phenomenon illustrated here (unexpectedly high DF in Chi-squared GOF tests) appears in Volume II of Kendall & Stuart, 5th edition. I am grateful for the opportunity afforded by this question to lead me back to this wonderful text, which is full of such useful analyses.

### Edit (Jan 2017)

Here is R code to produce the figure following "The standard wisdom about DF..."

#
# Simulate data, one iteration per column of x.
#
n <- 20
n.sim <- 1e4
bins <- qnorm(seq(0, 1, 1/4))
x <- matrix(rnorm(n*n.sim), nrow=n)
#
# Compute statistics.
#
m <- colMeans(x)
s <- apply(sweep(x, 2, m), 2, sd)
counts <- apply(matrix(as.numeric(cut(x, bins)), nrow=n), 2, tabulate, nbins=4)
expectations <- mapply(function(m,s) n*diff(pnorm(bins, m, s)), m, s)
chisquared <- colSums((counts - expectations)^2 / expectations)
#
# Plot histograms of means, variances, and chi-squared stats.  The first
# two confirm all is working as expected.
#
mfrow <- par("mfrow")
par(mfrow=c(1,3))
red <- "#a04040"  # Intended to show correct distributions
blue <- "#404090" # To show the putative chi-squared distribution
hist(m, freq=FALSE)
hist(s^2, freq=FALSE)
hist(chisquared, freq=FALSE, breaks=seq(0, ceiling(max(chisquared)), 1/4),
xlim=c(0, 13), ylim=c(0, 0.55),
col="#c0c0ff", border="#404040")
curve(ifelse(x <= 0, Inf, dchisq(x, df=2)), add=TRUE, col=red, lwd=2)
curve(ifelse(x <= 0, Inf, dchisq(x, df=1)), add=TRUE, col=blue, lwd=2)
par(mfrow=mfrow)

• This is an amazing answer. You win at the internet for this.
Dec 14 '11 at 3:00
• @caracal: as you know, ML methods for the original data are routine and widespread: for the normal distribution, for instance, the MLE of $\mu$ is the sample mean and the MLE of $\sigma$ is the square root of the sample standard deviation (without the usual bias correction). To obtain estimates based on counts, I computed the likelihood function for the counts--this requires computing values of the CDF at the cutpoints, taking their logs, multiplying by the counts, and adding up--and optimized it using generic optimization software.
– whuber
Dec 16 '11 at 14:35
• @caracal You probably no longer need it, but an example of R code for ML fitting of binned data now appears in a related question: stats.stackexchange.com/a/34894.
– whuber
Aug 22 '12 at 21:29
• Thanks for confirming me in not understanding the deeper meaning of DF wrt. statistics. I tried to read this up many times, and it never made any sense to me. Dec 3 '13 at 11:41
• @Clarinetist The principal point of my answer is to suggest that what you have been taught is based on a confusion of two concepts of DF. Although that confusion causes no problems for standard least-squares Normal-theory models, it leads to errors even in simple, common circumstances like analyses of contingency tables. That matrix rank gives the functional DF. In a least-squares linear model it happens to give the correct DF for certain kinds of tests, such as F tests. For the chi-squared test, the special conditions are enumerated later in the answer as points (1) and (2).
– whuber
Apr 28 '16 at 18:33

Or simply: the number of elements in a numerical array that you're allowed to change so that the value of the statistic remains unchanged.

# for instance if:
x + y + z = 10


you can change, for instance, x and y at random, but you cannot change z (you can, but not at random, therefore you're not free to change it - see Harvey's comment), 'cause you'll change the value of the statistic (Σ = 10). So, in this case df = 2.

• It is not quite correct to say "you cannot change z". In fact, you have to change z to make the sum equal 10. But you have no choice (no freedom) about what it changes to. You can change any two values, but not the third. Jul 28 '10 at 14:29

The concept is not at all difficult to make mathematical precise given a bit of general knowledge of $n$-dimensional Euclidean geometry, subspaces and orthogonal projections.

If $P$ is an orthogonal projection from $\mathbb{R}^n$ to a $p$-dimensional subspace $L$ and $x$ is an arbitrary $n$-vector then $Px$ is in $L$, $x - Px$ and $Px$ are orthogonal and $x - Px \in L^{\perp}$ is in the orthogonal complement of $L$. The dimension of this orthogonal complement, $L^{\perp}$, is $n-p$. If $x$ is free to vary in an $n$-dimensional space then $x - Px$ is free to vary in an $n-p$ dimensional space. For this reason we say that $x - Px$ has $n-p$ degrees of freedom.

These considerations are important to statistics because if $X$ is an $n$-dimensional random vector and $L$ is a model of its mean, that is, the mean vector $E(X)$ is in $L$, then we call $X-PX$ the vector of residuals, and we use the residuals to estimate the variance. The vector of residuals has $n-p$ degrees of freedom, that is, it is constrained to a subspace of dimension $n-p$.

If the coordinates of $X$ are independent and normally distributed with the same variance $\sigma^2$ then

• The vectors $PX$ and $X - PX$ are independent.
• If $E(X) \in L$ the distribution of the squared norm of the vector of residuals $||X - PX||^2$ is a $\chi^2$-distribution with scale parameter $\sigma^2$ and another parameter that happens to be the degrees of freedom $n-p$.

The sketch of proof of these facts is given below. The two results are central for the further development of the statistical theory based on the normal distribution. Note also that this is why the $\chi^2$-distribution has the parametrization it has. It is also a $\Gamma$-distribution with scale parameter $2\sigma^2$ and shape parameter $(n-p)/2$, but in the context above it is natural to parametrize in terms of the degrees of freedom.

I must admit that I don't find any of the paragraphs cited from the Wikipedia article particularly enlightening, but they are not really wrong or contradictory either. They say in an imprecise, and in a general loose sense, that when we compute the estimate of the variance parameter, but do so based on residuals, we base the computation on a vector that is only free to vary in a space of dimension $n-p$.

Beyond the theory of linear normal models the use of the concept of degrees of freedom can be confusing. It is, for instance, used in the parametrization of the $\chi^2$-distribution whether or not there is a reference to anything that could have any degrees of freedom. When we consider statistical analysis of categorical data there can be some confusion about whether the "independent pieces" should be counted before or after a tabulation. Furthermore, for constraints, even for normal models, that are not subspace constraints, it is not obvious how to extend the concept of degrees of freedom. Various suggestions exist typically under the name of effective degrees of freedom.

Before any other usages and meanings of degrees of freedom is considered I will strongly recommend to become confident with it in the context of linear normal models. A reference dealing with this model class is A First Course in Linear Model Theory, and there are additional references in the preface of the book to other classical books on linear models.

Proof of the results above: Let $\xi = E(X)$, note that the variance matrix is $\sigma^2 I$ and choose an orthonormal basis $z_1, \ldots, z_p$ of $L$ and an orthonormal basis $z_{p+1}, \ldots, z_n$ of $L^{\perp}$. Then $z_1, \ldots, z_n$ is an orthonormal basis of $\mathbb{R}^n$. Let $\tilde{X}$ denote the $n$-vector of the coefficients of $X$ in this basis, that is $$\tilde{X}_i = z_i^T X.$$ This can also be written as $\tilde{X} = Z^T X$ where $Z$ is the orthogonal matrix with the $z_i$'s in the columns. Then we have to use that $\tilde{X}$ has a normal distribution with mean $Z^T \xi$ and, because $Z$ is orthogonal, variance matrix $\sigma^2 I$. This follows from general linear transformation results of the normal distribution. The basis was chosen so that the coefficients of $PX$ are $\tilde{X}_i$ for $i= 1, \ldots, p$, and the coefficients of $X - PX$ are $\tilde{X}_i$ for $i= p+1, \ldots, n$. Since the coefficients are uncorrelated and jointly normal, they are independent, and this implies that $$PX = \sum_{i=1}^p \tilde{X}_i z_i$$ and $$X - PX = \sum_{i=p+1}^n \tilde{X}_i z_i$$ are independent. Moreover, $$||X - PX||^2 = \sum_{i=p+1}^n \tilde{X}_i^2.$$ If $\xi \in L$ then $E(\tilde{X}_i) = z_i^T \xi = 0$ for $i = p +1, \ldots, n$ because then $z_i \in L^{\perp}$ and hence $z_i \perp \xi$. In this case $||X - PX||^2$ is the sum of $n-p$ independent $N(0, \sigma^2)$-distributed random variables, whose distribution, by definition, is a $\chi^2$-distribution with scale parameter $\sigma^2$ and $n-p$ degrees of freedom.

• NRH, Thanks! (1) Why is $E(X)$ required to be inside $L$? (2) Why $PX$ and $X−PX$ are independent? (3) Is the dof in the random variable context defined from the dof in its deterministic case? For example, is the reason for $||X−PX||^2$ has dof $n-p$ because it is true when $X$ is a deterministic variable instead of a random variable? (4) Are there references (books, papers or links) that hold the same/similar opinion as yours?
– Tim
Oct 12 '11 at 22:50
• @Tim, $PX$ and $X-PX$ are independent, since they are normal and uncorrelated. Oct 13 '11 at 7:34
• @Tim, I have reworded the answer a little and given a proof of the stated results. The mean is required to be in $L$ to prove the result about the $\chi^2$-distribution. It is a model assumption. In the literature you should look for linear normal models or general linear models, but right now I can only recall some old, unpublished lecture notes. I will see if I can find a suitable reference.
– NRH
Oct 13 '11 at 11:06
• Wonderful answer. Thanks for the insight. One question: I got lost what you meant by the phrase "the mean vector $EX$ is in $L$". Can you explain? Are you try to define $E$? to define $L$? something else? Maybe this sentence is trying to do too much or be too concise for me. Can you elaborate what is the definition of $E$ in the context you mention: is it just $E(x_1,x_2,\dots,x_n) = (x_1+x_2+\dots+x_n)/n$? Can you elaborate on what is $L$ in this context (of normal iid coordinates)? Is it just $L = \mathbb{R}$?
– D.W.
Oct 13 '11 at 21:12
• @D.W. The $E$ is the expectation operator. So $E(X)$ is the vector of coordinatewise expectations of $X$. The subspace $L$ is any $p$-dimensional subspace of $\mathbb{R}^n$. It is a space of $n$-vectors and certainly not $\mathbb{R}$, but it can very well be one-dimensional. The simplest example is perhaps when it is spanned by the $\mathbf{1}$-vector with a 1 at all $n$-coordinates. This is the model of all coordinates of $X$ having the same mean value, but many more complicated models are possible.
– NRH
Oct 13 '11 at 22:02

It's really no different from the way the term "degrees of freedom" works in any other field. For example, suppose you have four variables: the length, the width, the area, and the perimeter of a rectangle. Do you really know four things? No, because there are only two degrees of freedom. If you know the length and the width, you can derive the area and the perimeter. If you know the length and the area, you can derive the width and the perimeter. If you know the area and the perimeter you can derive the length and the width (up to rotation). If you have all four, you can either say that the system is consistent (all of the variables agree with each other), or inconsistent (no rectangle could actually satisfy all of the conditions). A square is a rectangle with a degree of freedom removed; if you know any side of a square or its perimeter or its area, you can derive all of the others because there's only one degree of freedom.

In statistics, things get more fuzzy, but the idea is still the same. If all of the data that you're using as the input for a function are independent variables, then you have as many degrees of freedom as you have inputs. But if they have dependence in some way, such that if you had n - k inputs you could figure out the remaining k, then you've actually only got n - k degrees of freedom. And sometimes you need to take that into account, lest you convince yourself that the data are more reliable or have more predictive power than they really do, by counting more data points than you really have independent bits of data.

Moreover, all three definitions are almost trying to give a same message.

• Basically right, but I'm concerned that the middle paragraph could be read in a way that confuses correlation, independence (of random variables), and functional independence (of a manifold of parameters). The correlation-independence distinction is particularly important to maintain.
– whuber
Oct 12 '11 at 20:46
• @whuber: is it fine now? Oct 12 '11 at 20:55
• It's correct, but the way it uses terms would likely confuse some people. It still does not explicitly distinguish dependence of random variables from functional dependence. For example, the two variables in a (nondegenerate) bivariate normal distribution with nonzero correlation will be dependent (as random variables) but they still offer two degrees of freedom.
– whuber
Oct 12 '11 at 21:02
• This was copy-pasted from a reddit post I made in 2009. Apr 20 '14 at 3:22
• Our Help Center provide clear guidance on how to reference material written by others, so I hope the OP will come back to this post to take appropriate actions and engage in constructive interactions (we haven't seen him for a while, though).
– chl
Apr 24 '14 at 18:54

I really like first sentence from The Little Handbook of Statistical Practice. Degrees of Freedom Chapter

One of the questions an instrutor dreads most from a mathematically unsophisticated audience is, "What exactly is degrees of freedom?"

I think you can get really good understanding about degrees of freedom from reading this chapter.

• It would be nice to have an explanation for why degrees of freedom is important, rather than just what it is. For instance, showing that the estimate of variance with 1/n is biased but using 1/(n-1) yields an unbiased estimator. Jul 31 '10 at 20:12
• "I think you can get really good understanding about degrees of freedom from reading this chapter" - Definitely not. Oct 6 '20 at 18:18

Wikipedia asserts that degrees of freedom of a random vector can be interpreted as the dimensions of the vector subspace. I want to go step-by-step, very basically through this as a partial answer and elaboration on the Wikipedia entry.

The example proposed is that of a random vector corresponding to the measurements of a continuous variable for different subjects, expressed as a vector extending from the origin $[a\,b\,c]^T$. Its orthogonal projection on the vector $[1\,1\,1]^T$ results in a vector equal to the projection of the vector of measurement means ($\bar{x}=1/3(a+b+c)$), i.e. $[\bar x \, \bar x \, \bar x]^T$, dotted with the $\vec{1}$ vector, $[1\,1\,1]^T$ This projection onto the subspace spanned by the vector of ones has $1\,\text{degree of freedom}$. The residual vector (distance from the mean) is the least-squares projection onto the $(n − 1)$-dimensional orthogonal complement of this subspace, and has $n − 1\,\text{degrees of freedom}$, $n$ being the total number of components of the vector (in our case $3$ since we are in $\mathbb{R}^3$ in the example).This can be simply proven by obtaining the dot product of $[\bar{x}\,\bar{x}\,\bar{x}]^T$ with the difference between $[a\,b\,c]^T$ and $[\bar{x}\,\bar{x}\,\bar{x}]^T$:

$$[\bar{x}\, \bar{x}\,\bar{x}]\, \begin{bmatrix} a-\bar{x}\\b-\bar{x}\\c-\bar{x}\end{bmatrix}=$$

$$= \bigg[\tiny\frac{(a+b+c)}{3}\, \bigg(a-\frac{(a+b+c)}{3}\bigg)\bigg]+ \bigg[\tiny\frac{(a+b+c)}{3} \,\bigg(b-\frac{(a+b+c)}{3}\bigg)\bigg]+ \bigg[\tiny\frac{(a+b+c)}{3} \,\bigg(c-\frac{(a+b+c)}{3}\bigg)\bigg]$$

$$=\tiny \frac{(a+b+c)}{3}\bigg[ \bigg(\tiny a-\frac{(a+b+c)}{3}\bigg)+ \bigg(b-\frac{(a+b+c)}{3}\bigg)+ \bigg(c-\frac{(a+b+c)}{3}\bigg)\bigg]$$

$$= \tiny \frac{(a+b+c)}{3}\bigg[\tiny \frac{1}{3} \bigg(\tiny 3a-(a+b+c)+ 3b-(a+b+c)+3c-(a+b+c)\bigg)\bigg]$$

$$=\tiny\frac{(a+b+c)}{3}\bigg[\tiny\frac{1}{3} (3a-3a+ 3b-3b+3c-3c)\bigg]\large= 0$$.

And this relationship extends to any point in a plane orthogonal to $[\bar{x}\,\bar{x}\,\bar{x}]^T$. This concept is important in understanding why $\frac 1 {\sigma^2} \Big((X_1-\bar X)^2 + \cdots + (X_n - \bar X)^2 \Big) \sim \chi^2_{n-1}$, a step in the derivation of the t-distribution(here and here).

Let's take the point $[35\,50\,80]^T$, corresponding to three observations. The mean is $55$, and the vector $[55\,\,55\,\,55]^T$ is the normal (orthogonal) to a plane, $55x + 55y + 55z = D$. Plugging in the point coordinates into the plane equation, $D = -9075$.

Now we can choose any other point in this plane, and the mean of its coordinates is going to be $55$, geometrically corresponding to its projection onto the vector $[1\,\,1\,\,1]^T$. Hence for every mean value (in our example, $55$) we can choose an infinite number of pairs of coordinates in $\mathbb{R}^2$ without restriction ($2\,\text{degrees of freedom}$); yet, since the plane is in $\mathbb{R}^3$, the third coordinate will come determined by the equation of the plane (or, geometrically the orthogonal projection of the point onto $[55\,\,55\,\,55]^T$.

Here is representation of three points (in white) lying on the plane (cerulean blue) orthogonal to $[55\,\,55\,\,55]^T$ (arrow): $[35\,\,50\,\,80]^T$, $[80\,\,80\,\,5]$ and $[90\,\,15\,\,60]$ all of them on the plane (subspace with $2\,\text{df}$), and then with a mean of their components of $55$, and an orthogonal projection to $[1\,\,1\,\,1]^T$ (subspace with $1\,\text{df}$) equal to $[55\,\,55\,\,55]^T$:

In my classes, I use one "simple" situation that might help you wonder and perhaps develop a gut feeling for what a degree of freedom may mean.

It is kind of a "Forrest Gump" approach to the subject, but it is worth the try.

Consider you have 10 independent observations $X_1, X_2, \ldots, X_{10}\sim N(\mu,\sigma^2)$ that came right from a normal population whose mean $\mu$ and variance $\sigma^2$ are unknown.

Your observations bring to you collectively information both about $\mu$ and $\sigma^2$. After all, your observations tend to be spread around one central value, which ought to be close to the actual and unknown value of $\mu$ and, likewise, if $\mu$ is very high or very low, then you can expect to see your observations gather around a very high or very low value respectively. One good "substitute" for $\mu$ (in the absence of knowledge of its actual value) is $\bar X$, the average of your observation.

Also, if your observations are very close to one another, that is an indication that you can expect that $\sigma^2$ must be small and, likewise, if $\sigma^2$ is very large, then you can expect to see wildly different values for $X_1$ to $X_{10}$.

If you were to bet your week's wage on which should be the actual values of $\mu$ and $\sigma^2$, you would need to choose a pair of values in which you would bet your money. Let's not think of anything as dramatic as losing your paycheck unless you guess $\mu$ correctly until its 200th decimal position. Nope. Let's think of some sort of prizing system that the closer you guess $\mu$ and $\sigma^2$ the more you get rewarded.

In some sense, your better, more informed, and more polite guess for $\mu$'s value could be $\bar X$. In that sense, you estimate that $\mu$ must be some value around $\bar X$. Similarly, one good "substitute" for $\sigma^2$ (not required for now) is $S^2$, your sample variance, which makes a good estimate for $\sigma$.

If your were to believe that those substitutes are the actual values of $\mu$ and $\sigma 2$, you would probably be wrong, because very slim are the chances that you were so lucky that your observations coordinated themselves to get you the gift of $\bar X$ being equal to $\mu$ and $S^2$ equal to $\sigma^2$. Nah, probably it didn't happen.

But you could be at different levels of wrong, varying from a bit wrong to really, really, really miserably wrong (a.k.a., "Bye-bye, paycheck; see you next week!").

Ok, let's say that you took $\bar X$ as your guess for $\mu$. Consider just two scenarios: $S^2=2$ and $S^2=20,000,000$. In the first, your observations sit pretty and close to one another. In the latter, your observations vary wildly. In which scenario you should be more concerned with your potential losses? If you thought of the second one, you're right. Having a estimate about $\sigma^2$ changes your confidence on your bet very reasonably, for the larger $\sigma^2$ is, the wider you can expect $\bar X$ to variate.

But, beyond information about $\mu$ and $\sigma^2$, your observations also carry some amount of just pure random fluctuation that is not informative neither about $\mu$ nor about $\sigma^2$.

How can you notice it?

Well, let's assume, for sake of argument, that there is a God and that He has spare time enough to give Himself the frivolity of telling you specifically the real (and so far unknown) values of both $\mu$ and $\sigma$.

And here is the annoying plot twist of this lysergic tale: He tells it to you after you placed your bet. Perhaps to enlighten you, perhaps to prepare you, perhaps to mock you. How could you know?

Well, that makes the information about $\mu$ and $\sigma^2$ contained in your observations quite useless now. Your observations' central position $\bar X$ and variance $S^2$ are no longer of any help to get closer to the actual values of $\mu$ and $\sigma^2$, for you already know them.

One of the benefits of your good acquaintance with God is that you actually know by how much you failed to guess correctly $\mu$ by using $\bar X$, that is, $(\bar X - \mu)$ your estimation error.

Well, since $X_i\sim N(\mu,\sigma^2)$, then $\bar X\sim N(\mu,\sigma^2/10)$ (trust me in that if you will), also $(\bar X - \mu)\sim N(0,\sigma^2/10)$ (ok, trust me in that on too) and, finally, $$\frac{\bar X - \mu}{\sigma/\sqrt{10}} \sim N(0,1)$$ (guess what? trust me in that one as well), which carries absolutely no information about $\mu$ or $\sigma^2$.

You know what? If you took any of your individual observations as a guess for $\mu$, your estimation error $(X_i-\mu)$ would be distributed as $N(0,\sigma^2)$. Well, between estimating $\mu$ with $\bar X$ and any $X_i$, choosing $\bar X$ would be better business, because $Var(\bar X) = \sigma^2/10 < \sigma^2 = Var(X_i)$, so $\bar X$ was less prone to be astray from $\mu$ than an individual $X_i$.

Anyway, $(X_i-\mu)/\sigma\sim N(0,1)$ is also absolutely non informative about neither $\mu$ nor $\sigma^2$.

"Will this tale ever end?" you may be thinking. You also may be thinking "Is there any more random fluctuation that is non informative about $\mu$ and $\sigma^2$?".

[I prefer to think that you are thinking of the latter.]

Yes, there is!

The square of your estimation error for $\mu$ with $X_i$ divided by $\sigma$, $$\frac{(X_i-\mu)^2}{\sigma^2} = \left(\frac{X_i-\mu}{\sigma}\right)^2 \sim \chi^2$$ has a Chi-squared distribution, which is the distribution of the square $Z^2$ of a standard Normal $Z\sim N(0,1)$, which I am sure you noticed has absolutely no information about either $\mu$ nor $\sigma^2$, but conveys information about the variability you should expect to face.

That is a very well known distribution that arises naturally from the very scenario of you gambling problem for every single one of your ten observations and also from your mean: $$\frac{(\bar X-\mu)^2}{\sigma^2/10} = \left(\frac{\bar X-\mu}{\sigma/\sqrt{10}}\right)^2 = \left(N(0,1)\right)^2 \sim\chi^2$$ and also from the gathering of your ten observations' variation: $$\sum_{i=1}^{10} \frac{(X_i-\mu)^2}{\sigma^2/10} =\sum_{i=1}^{10} \left(\frac{X_i-\mu}{\sigma/\sqrt{10}}\right)^2 =\sum_{i=1}^{10} \left(N(0,1)\right)^2 =\sum_{i=1}^{10} \chi^2.$$ Now that last guy doesn't have a Chi-squared distribution, because he is the sum of ten of those Chi-squared distributions, all of them independent from one another (because so are $X_1, \ldots, X_{10}$). Each one of those single Chi-squared distribution is one contribution to the amount of random variability you should expect to face, with roughly the same amount of contribution to the sum.

The value of each contribution is not mathematically equal to the other nine, but all of them have the same expected behavior in distribution. In that sense, they are somehow symmetric.

Each one of those Chi-square is one contribution to the amount of pure, random variability you should expect in that sum.

If you had 100 observations, the sum above would be expected to be bigger just because it have more sources of contibutions.

Each of those "sources of contributions" with the same behavior can be called degree of freedom.

Now take one or two steps back, re-read the previous paragraphs if needed to accommodate the sudden arrival of your quested-for degree of freedom.

Yep, each degree of freedom can be thought of as one unit of variability that is obligatorily expected to occur and that brings nothing to the improvement of guessing of $\mu$ or $\sigma^2$.

The thing is, you start to count on the behavior of those 10 equivalent sources of variability. If you had 100 observations, you would have 100 independent equally-behaved sources of strictly random fluctuation to that sum.

That sum of 10 Chi-squares gets called a Chi-squared distributions with 10 degrees of freedom from now on, and written $\chi^2_{10}$. We can describe what to expect from it starting from its probability density function, that can be mathematically derived from the density from that single Chi-squared distribution (from now on called Chi-squared distribution with one degree of freedom and written $\chi^2_1$), that can be mathematically derived from the density of the normal distribution.

"So what?" --- you might be thinking --- "That is of any good only if God took the time to tell me the values of $\mu$ and $\sigma^2$, of all the things He could tell me!"

Indeed, if God Almighty were too busy to tell you the values of $\mu$ and $\sigma^2$, you would still have that 10 sources, that 10 degrees of freedom.

Things start to get weird (Hahahaha; only now!) when you rebel against God and try and get along all by yourself, without expecting Him to patronize you.

You have $\bar X$ and $S^2$, estimators for $\mu$ and $\sigma^2$. You can find your way to a safer bet.

You could consider calculating the sum above with $\bar X$ and $S^2$ in the places of $\mu$ and $\sigma^2$: $$\sum_{i=1}^{10} \frac{(X_i-\bar X)^2}{S^2/10} =\sum_{i=1}^{10} \left(\frac{X_i-\bar X}{S/\sqrt{10}}\right)^2,$$ but that is not the same as the original sum.

"Why not?" The term inside the square of both sums are very different. For instance, it is unlikely but possible that all your observations end up being larger than $\mu$, in which case $(X_i-\mu) > 0$, which implies $\sum_{i=1}^{10}(X_i-\mu) > 0$, but, by its turn, $\sum_{i=1}^{10}(X_i-\bar X) = 0$, because $\sum_{i=1}^{10}X_i-10 \bar X =10 \bar X - 10 \bar X = 0$.

Worse, you can prove easily (Hahahaha; right!) that $\sum_{i=1}^{10}(X_i-\bar X)^2 \le \sum_{i=1}^{10}(X_i-\mu)^2$ with strict inequality when at least two observations are different (which is not unusual).

"But wait! There's more!" $$\frac{X_i-\bar X}{S/\sqrt{10}}$$ doesn't have standard normal distribution, $$\frac{(X_i-\bar X)^2}{S^2/10}$$ doesn't have Chi-squared distribution with one degree of freedom, $$\sum_{i=1}^{10} \frac{(X_i-\bar X)^2}{S^2/10}$$ doesn't have Chi-squared distribution with 10 degrees of freedom $$\frac{\bar X-\mu}{S/\sqrt{10}}$$ doesn't have standard normal distribution.

"Was it all for nothing?"

No way. Now comes the magic! Note that $$\sum_{i=1}^{10} \frac{(X_i-\bar X)^2}{\sigma^2} =\sum_{i=1}^{10} \frac{[X_i-\mu+\mu-\bar X]^2}{\sigma^2} =\sum_{i=1}^{10} \frac{[(X_i-\mu)-(\bar X-\mu)]^2}{\sigma^2} =\sum_{i=1}^{10} \frac{(X_i-\mu)^2-2(X_i-\mu)(\bar X-\mu)+(\bar X-\mu)^2}{\sigma^2} =\sum_{i=1}^{10} \frac{(X_i-\mu)^2-(\bar X-\mu)^2}{\sigma^2} =\sum_{i=1}^{10} \frac{(X_i-\mu)^2}{\sigma^2}-\sum_{i=1}^{10} \frac{(\bar X-\mu)^2}{\sigma^2} =\sum_{i=1}^{10} \frac{(X_i-\mu)^2}{\sigma^2}-10\frac{(\bar X-\mu)^2}{\sigma^2} =\sum_{i=1}^{10} \frac{(X_i-\mu)^2}{\sigma^2}-\frac{(\bar X-\mu)^2}{\sigma^2/10}$$ or, equivalently, $$\sum_{i=1}^{10} \frac{(X_i-\mu)^2}{\sigma^2} =\sum_{i=1}^{10} \frac{(X_i-\bar X)^2}{\sigma^2} +\frac{(\bar X-\mu)^2}{\sigma^2/10}.$$ Now we get back to those known faces.

The first term has Chi-squared distribution with 10 degrees of freedom and the last term has Chi-squared distribution with one degree of freedom(!).

We simply split a Chi-square with 10 independent equally-behaved sources of variability in two parts, both positive: one part is a Chi-square with one source of variability and the other we can prove (leap of faith? win by W.O.?) to be also a Chi-square with 9 (= 10-1) independent equally-behaved sources of variability, with both parts independent from one another.

This is already a good news, since now we have its distribution.

Alas, it uses $\sigma^2$, to which we have no access (recall that God is amusing Himself on watching our struggle).

Well, $$S^2=\frac{1}{10-1}\sum_{i=1}^{10} (X_i-\bar X)^2,$$ so $$\sum_{i=1}^{10} \frac{(X_i-\bar X)^2}{\sigma^2} =\frac{\sum_{i=1}^{10} (X_i-\bar X)^2}{\sigma^2} =\frac{(10-1)S^2}{\sigma^2} \sim\chi^2_{(10-1)}$$ therefore $$\frac{\bar X-\mu}{S/\sqrt{10}} =\frac{\frac{\bar X-\mu}{\sigma/\sqrt{10}}}{\frac{S}{\sigma}} =\frac{\frac{\bar X-\mu}{\sigma/\sqrt{10}}}{\sqrt{\frac{S^2}{\sigma^2}}} =\frac{\frac{\bar X-\mu}{\sigma/\sqrt{10}}}{\sqrt{\frac{\frac{(10-1)S^2}{\sigma^2}}{(10-1)}}} =\frac{N(0,1)}{\sqrt{\frac{\chi^2_{(10-1)}}{(10-1)}}},$$ which is a distribution that is not the standard normal, but whose density can be derived from the densities of the standard normal and the Chi-squared with $(10-1)$ degrees of freedom.

One very, very smart guy did that math[^1] in the beginning of 20th century and, as an unintended consequence, he made his boss the absolute world leader in the industry of Stout beer. I am talking about William Sealy Gosset (a.k.a. Student; yes, that Student, from the $t$ distribution) and Saint James's Gate Brewery (a.k.a. Guinness Brewery), of which I am a devout.

[^1]: @whuber told in the comments below that Gosset did not do the math, but guessed instead! I really don't know which feat is more surprising for that time.

That, my dear friend, is the origin of the $t$ distribution with $(10-1)$ degrees of freedom. The ratio of a standard normal and the squared root of an independent Chi-square divided by its degrees of freedom, which, in an unpredictable turn of tides, wind up describing the expected behavior of the estimation error you undergo when using the sample average $\bar X$ to estimate $\mu$ and using $S^2$ to estimate the variability of $\bar X$.

There you go. With an awful lot of technical details grossly swept behind the rug, but not depending solely on God's intervention to dangerously bet your whole paycheck.

• Thank you for such an effort! I confess that I found your explanation less than convincing, though. It seems to founder at this crucial junction: "Each of those "sources of contributions" with the same behavior can be called degree of freedom." If you had instead summed $10$ independent normal variates rather than $10$ independent chi-squared variates, you would end up with--one normal variate. Somehow the "degrees of freedom" get completely swallowed up. Evidently there is something special about chi-squared you haven't yet described. BTW, Gosset didn't do the math: he guessed!
– whuber
Jan 1 '17 at 3:33
• Thank you very much for your evaluation, @whuber! It's amazing how many typos pop up once you forgot what you wrote. About your evaluation, I intended just to illustrate another way of thinking -- a little bit less mathematical in some sense. Also, I am not grasping fully what you meant with If you had instead summed 10 independent normal variates rather than 10 independent chi-squared variates, you would end up with--one normal variate -- which I guessed to hold your key-point. I will try to elaborate about it, hoping to improve the post. Jan 2 '17 at 15:08

This particular issue is quite frustrating for students in statistics courses, since they often cannot get a straight answer on exactly what a degree-of-freedom is defined to be. I will try to clear that up here. Suppose we have a random vector $$\mathbf{x} \in \mathbb{R}^n$$ and we form a new random vector $$\mathbf{t} = T(\mathbf{x})$$ via the linear function $$T$$. Formally, the degrees-of-freedom of $$\mathbf{t}$$ is the dimension of the space of allowable values for this vector, which is:

$$DF \equiv \dim \mathscr{T} \equiv \dim \{ \mathbf{t} = T(\mathbf{x}) | \mathbf{x} \in \mathbb{R}^n \}.$$

The initial random vector $$\mathbf{x}$$ has an allowable space of dimension $$n$$, so it has $$n$$ degrees of freedom. Often the function $$T$$ will reduce the dimension of the allowable space of outcomes, and so $$\mathbf{t}$$ may have a lower degrees-of-freedom than $$\mathbf{x}$$. For example, in an answer to a related question you can see this formal definition of the degrees-of-freedom being used to explain Bessel's correction in the sample variance formula. In that particular case, transforming an initial sample to obtain its deviations from the sample mean leads to a deviation vector that has $$n-1$$ degrees-of-freedom (i.e., it is a vector in an allowable space with dimension $$n-1$$).

When you apply this formal definition to statistical problems, you will usually find that the imposition of a single "constraint" on the random vector (via a linear equation on that vector) reduces the dimension of its allowable values by one, and thus reduces the degrees-of-freedom by one. As such, you will find that the above formal definition corresponds with the informal explanations you have been given.

In undergraduate courses on statistics, you will generally find a lot of hand-waving and informal explanation of degrees-of-freedom, often via analogies or examples. The reason for this is that the formal definition requires an understanding of vector algebra and the geometry of vector spaces, which may be lacking in introductory statistics courses at an undergraduate level.

You can see the degree of freedom as the number of observations minus the number of necessary relations among these observations. By exemple if you have $$n$$ sample of independant normal distribution observations $$X_1,\dots,X_n$$. The random variable $$\sum_{i=1}^n (X_i-\overline{X}_n)^2\sim \mathcal{X}^2_{n-1}$$, where $$\overline{X}_n = \frac{1}{n}\sum_{i=1}^n X_i$$. The degree of freedom here is $$n-1$$ because, their is one necessary relation between theses observations $$(\overline{X}_n = \frac{1}{n}\sum_{i=1}^n X_i)$$.

An intuitive explanation of degrees of freedom is that they represent the number of independent pieces of information available in the data for estimating a parameter (i.e., unknown quantity) of interest.

As an example, in a simple linear regression model of the form:

$$Y_i=\beta_0 + \beta_1\cdot X_i + \epsilon_i,\quad i=1,\ldots, n$$

where the $$\epsilon_i$$'s represent independent normally distributed error terms with mean 0 and standard deviation $$\sigma$$, we use 1 degree of freedom to estimate the intercept $$\beta_0$$ and 1 degree of freedom to estimate the slope $$\beta_1$$. Since we started out with $$n$$ observations and used up 2 degrees of freedom (i.e., two independent pieces of information), we are left with $$n-2$$ degrees of freedom (i.e., $$n-2$$ independent pieces of information) available for estimating the error standard deviation $$\sigma$$.

• Thanks very much for your edits to my answer, @COOLSerdash! May 8 '19 at 23:06

The clearest "formal" definition of degrees-of-freedom is that it is the dimension of the space of allowable values for a random vector. This generally arises in a context where we have a sample vector $$\mathbf{x} \in \mathbb{R}^n$$ and we form a new random vector $$\mathbf{t} = T(\mathbf{x})$$ via the linear function $$T$$. Formally, the degrees-of-freedom of $$\mathbf{t}$$ is the dimension of the space of allowable values for this vector, which is:

$$DF \equiv \dim \mathscr{T} \equiv \dim \{ \mathbf{t} = T(\mathbf{x}) | \mathbf{x} \in \mathbb{R}^n \}.$$

If we represent this linear transformation by the matrix transformation $$T(\mathbf{x}) = \mathbf{T} \mathbf{x}$$ then we have:

\begin{aligned} DF &= \dim \{ \mathbf{t} = T(\mathbf{x}) | \mathbf{x} \in \mathbb{R}^n \} \\[6pt] &= \dim \{ \mathbf{T} \mathbf{x} | \mathbf{x} \in \mathbb{R}^n \} \\[6pt] &= \text{rank} \ \mathbf{T} \\[6pt] &= n - \text{Ker} \ \mathbf{T}, \\[6pt] \end{aligned}

where the last step follows from the rank-nullity theorem. This means that when we transform $$\mathbf{x}$$ by the linear transformation $$T$$ we lose degrees-of-freedom equal to the kernel (nullspace) of $$\mathbf{T}$$. In statistical problems, there is a close relationship between the eigenvalues of $$\mathbf{T}$$ and the loss of degrees-of-freedom from the transformation. Often the loss of degrees-of-freedom is equivalent to the number of zero eigenvalues in the transformation matrix $$\mathbf{T}$$.

For example, in this answer we see that Bessel's correction to the sample variance, adjusting for the degrees-of-freedom of the vector of deviations from the mean, is closely related to the eigenvalues of the centering matrix. An identical result occurs in higher dimensions in linear regression analysis. In other statistical problems, similar relationships occur between the eigenvalues of the transformation matrix and the loss of degrees-of-freedom.

The above result also formalises the notation that one loses a degree-of-freedom for each "constraint" imposed on the observable vector of interest. Thus, in simple univariate sampling problems, when looking at the sample variance, one loses a degree-of-freedom from estimating the mean. In linear regression models, when looking at the MSE, one loses a degree-of-freedom for each model coefficient that was estimated.

For me the first explanation I understood was:

If you know some statistical value like mean or variation, how many variables of data you need to know before you can know the value of every variable?

This is the same as aL3xa said, but without giving any data point a special role and close to the third case given in the answer. In this way the same example would be:

If you know the mean of data, you need to know the values for all but one data point, to know the value to all data points.

• Variables --> observations Mar 30 '19 at 12:25

Think of it this way. Variances are additive when independent. For example, suppose we are throwing darts at a board and we measure the standard deviations of the $x$ and $y$ displacements from the exact center of the board. Then $V_{x,y}=V_x+V_y$. But, $V_x=SD_x^2$ if we take the square root of the $V_{x,y}$ formula, we get the distance formula for orthogonal coordinates, $SD_{x,y}=\sqrt{SD_x^2+SD_y^2}$. Now all we have to show is that standard deviation is a representative measure of displacement away from the center of the dart board. Since $SD_x=\sqrt{\dfrac{\sum_{i=1}^n(x_i-\bar{x})^2}{n-1}}$, we have a ready means of discussing df. Note that when $n=1$, then $x_1-\bar{x}=0$ and the ratio $\dfrac{\sum_{i=1}^n(x_i-\bar{x})^2}{n-1}\rightarrow \dfrac{0}{0}$. In other words, there is no deviation to be had between one dart's $x$-coordinate and itself. The first time we have a deviation is for $n=2$ and there is only one of them, a duplicate. That duplicate deviation is the squared distance between $x_1$ or $x_2$ and $\bar{x}=\dfrac{x_1+x_2}{2}$ because $\bar{x}$ is the midpoint between or average of $x_1$ and $x_2$. In general, for $n$ distances we remove 1 because $\bar{x}$ is dependent on all $n$ of those distances. Now, $n-1$ represents the degrees of freedom because it normalizes for the number of unique outcomes to make an expected square distance. when divided into the sum of those square distances.