Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

From Wikipedia, there are three interpretations of the degrees of freedom of a statistic:

In statistics, the number of degrees of freedom is the number of values in the final calculation of a statistic that are free to vary.

Estimates of statistical parameters can be based upon different amounts of information or data. The number of independent pieces of information that go into the estimate of a parameter is called the degrees of freedom (df). In general, the degrees of freedom of an estimate of a parameter is equal to the number of independent scores that go into the estimate minus the number of parameters used as intermediate steps in the estimation of the parameter itself (which, in sample variance, is one, since the sample mean is the only intermediate step).

Mathematically, degrees of freedom is the dimension of the domain of a random vector, or essentially the number of 'free' components: how many components need to be known before the vector is fully determined.

The bold words are what I don't quite understand. If possible, some mathematical formulations will help clarify the concept.

Also do the three interpretations agree with each other?

share|improve this question
    
Check out this explanation – George Dontas Jul 28 '10 at 10:26
2  
Also see this question "What are degrees of freedom?" – Jeromy Anglim Oct 12 '11 at 22:12
1  
@Jeromy: Seen already. The replies and links there seem not formal enough, and too localized to some particular examples. – Tim Oct 12 '11 at 22:23
1  
Yep. I agree they're different questions. I just wanted create a link between the two. – Jeromy Anglim Oct 12 '11 at 22:31
up vote 132 down vote accepted

This is a subtle question. It takes a thoughtful person not to understand those quotations! Although they are suggestive, it turns out that none of them is exactly or generally correct. I haven't the time (and there isn't the space here) to give a full exposition, but I would like to share one approach and an insight that it suggests.

Where does the concept of degrees of freedom (DF) arise? The contexts in which it's found in elementary treatments are:

  • The Student t-test and its variants such as the Welch or Satterthwaite solutions to the Behrens-Fisher problem (where two populations have different variances).

  • The Chi-squared distribution (defined as a sum of squares of independent standard Normals), which is implicated in the sampling distribution of the variance.

  • The F-test (of ratios of estimated variances).

  • The Chi-squared test, comprising its uses in (a) testing for independence in contingency tables and (b) testing for goodness of fit of distributional estimates.

In spirit, these tests run a gamut from being exact (the Student t-test and F-test for Normal variates) to being good approximations (the Student t-test and the Welch/Satterthwaite tests for not-too-badly-skewed data) to being based on asymptotic approximations (the Chi-squared test). An interesting aspect of some of these is the appearance of non-integral "degrees of freedom" (the Welch/Satterthwaite tests and, as we will see, the Chi-squared test). This is of especial interest because it is the first hint that DF is not any of the things claimed of it.

We can dispose right away of some of the claims in the question. Because "final calculation of a statistic" is not well-defined (it apparently depends on what algorithm one uses for the calculation), it can be no more than a vague suggestion and is worth no further criticism. Similarly, neither "number of independent scores that go into the estimate" nor "the number of parameters used as intermediate steps" are well-defined.

"Independent pieces of information that go into [an] estimate" is difficult to deal with, because there are two different but intimately related senses of "independent" that can be relevant here. One is independence of random variables; the other is functional independence. As an example of the latter, suppose we collect morphometric measurements of subjects--say, for simplicity, the three side lengths $X$, $Y$, $Z$, surface areas $S=2(XY+YZ+ZX)$, and volumes $V=XYZ$ of a set of wooden blocks. The three side lengths can be considered independent random variables, but all five variables are dependent RVs. The five are also functionally dependent because the codomain (not the "domain"!) of the vector-valued random variable $(X,Y,Z,S,V)$ traces out a three-dimensional manifold in $\mathbb{R}^5$. (Thus, locally at any point $\omega\in\mathbb{R}^5$, there are two functions $f_\omega$ and $g_\omega$ for which $f_\omega(X(\psi),\ldots,V(\psi))=0$ and $g_\omega(X(\psi),\ldots,V(\psi))=0$ for points $\psi$ "near" $\omega$ and the derivatives of $f$ and $g$ evaluated at $\omega$ are linearly independent.) However--here's the kicker--for many probability measures on the blocks, subsets of the variables such as $(X,S,V)$ are dependent as random variables but functionally independent.

Having been alerted by these potential ambiguities, let's hold up the Chi-squared goodness of fit test for examination, because (a) it's simple, (b) it's one of the common situations where people really do need to know about DF to get the p-value right and (c) it's often used incorrectly. Here's a brief synopsis of the least controversial application of this test:

  • You have a collection of data values $(x_1, \ldots, x_n)$, considered as a sample of a population.

  • You have estimated some parameters $\theta_1, \ldots, \theta_p$ of a distribution. For example, you estimated the mean $\theta_1$ and standard deviation $\theta_2 = \theta_p$ of a Normal distribution, hypothesizing that the population is normally distributed but not knowing (in advance of obtaining the data) what $\theta_1$ or $\theta_2$ might be.

  • In advance, you created a set of $k$ "bins" for the data. (It may be problematic when the bins are determined by the data, even though this is often done.) Using these bins, the data are reduced to the set of counts within each bin. Anticipating what the true values of $(\theta)$ might be, you have arranged it so (hopefully) each bin will receive approximately the same count. (Equal-probability binning assures the chi-squared distribution really is a good approximation to the true distribution of the chi-squared statistic about to be described.)

  • You have a lot of data--enough to assure that almost all bins ought to have counts of 5 or greater. (This, we hope, will enable the sampling distribution of the $\chi^2$ statistic to be approximated adequately by some $\chi^2$ distribution.)

Using the parameter estimates, you can compute the expected count in each bin. The Chi-squared statistic is the sum of the ratios

$$\frac{(\text{observed}-\text{expected})^2}{\text{expected}}.$$

This, many authorities tell us, should have (to a very close approximation) a Chi-squared distribution. But there's a whole family of such distributions. They are differentiated by a parameter $\nu$ often referred to as the "degrees of freedom." The standard reasoning about how to determine $\nu$ goes like this

I have $k$ counts. That's $k$ pieces of data. But there are (functional) relationships among them. To start with, I know in advance that the sum of the counts must equal $n$. That's one relationship. I estimated two (or $p$, generally) parameters from the data. That's two (or $p$) additional relationships, giving $p+1$ total relationships. Presuming they are all (functionally) independent, that leaves only $k-p-1$ (functionally) independent "degrees of freedom": that's the value to use for $\nu$.

The problem with this reasoning (which is the sort of calculation the quotations in the question are hinting at) is that it's wrong except when some special additional conditions hold. Moreover, those conditions have nothing to do with independence (functional or statistical), with numbers of "components" of the data, with the numbers of parameters, nor with anything else referred to in the original question.

Let me show you with an example. (To make it as clear as possible, I'm using a small number of bins, but that's not essential.) Let's generate 20 independent and identically distributed (iid) standard Normal variates and estimate their mean and standard deviation with the usual formulas (mean = sum/count, etc.). To test goodness of fit, create four bins with cutpoints at the quartiles of a standard normal: -0.675, 0, +0.657, and use the bin counts to generate a Chi-squared statistic. Repeat as patience allows; I had time to do 10,000 repetitions.

The standard wisdom about DF says we have 4 bins and 1+2 = 3 constraints, implying the distribution of these 10,000 Chi-squared statistics should follow a Chi-squared distribution with 1 DF. Here's the histogram:

Figure 1

The dark blue line graphs the PDF of a $\chi^2(1)$ distribution--the one we thought would work--while the dark red line graphs that of a $\chi^2(2)$ distribution (which would be a good guess if someone were to tell you that $\nu=1$ is incorrect). Neither fits the data.

You might expect the problem to be due to the small size of the data sets ($n$=20) or perhaps the small size of the number of bins. However, the problem persists even with very large datasets and larger numbers of bins: it is not merely a failure to reach an asymptotic approximation.

Things went wrong because I violated two requirements of the Chi-squared test:

  1. You must use the Maximum Likelihood estimate of the parameters. (This requirement can, in practice, be slightly violated.)

  2. You must base that estimate on the counts, not on the actual data! (This is crucial.)

Figure 2

The red histogram depicts the chi-squared statistics for 10,000 separate iterations, following these requirements. Sure enough, it visibly follows the $\chi^2(1)$ curve (with an acceptable amount of sampling error), as we had originally hoped.

The point of this comparison--which I hope you have seen coming--is that the correct DF to use for computing the p-values depends on many things other than dimensions of manifolds, counts of functional relationships, or the geometry of Normal variates. There is a subtle, delicate interaction between certain functional dependencies, as found in mathematical relationships among quantities, and distributions of the data, their statistics, and the estimators formed from them. Accordingly, it cannot be the case that DF is adequately explainable in terms of the geometry of multivariate normal distributions, or in terms of functional independence, or as counts of parameters, or anything else of this nature.

We are led to see, then, that "degrees of freedom" is merely a heuristic that suggests what the sampling distribution of a (t, Chi-squared, or F) statistic ought to be, but it is not dispositive. Belief that it is dispositive leads to egregious errors. (For instance, the top hit on Google when searching "chi squared goodness of fit" is a Web page from an Ivy League university that gets most of this completely wrong! In particular, a simulation based on its instructions shows that the chi-squared value it recommends as having 7 DF actually has 9 DF.)

With this more nuanced understanding, it's worthwhile to re-read the Wikipedia article in question: in its details it gets things right, pointing out where the DF heuristic tends to work and where it is either an approximation or does not apply at all.


A good account of the phenomenon illustrated here (unexpectedly high DF in Chi-squared GOF tests) appears in Volume II of Kendall & Stuart, 5th edition. I am grateful for the opportunity afforded by this question to lead me back to this wonderful text, which is full of such useful analyses.

share|improve this answer
22  
This is an amazing answer. You win at the internet for this. – Adam Dec 14 '11 at 3:00
    
Great answer! I'm just reproducing the $\chi^2$ simulation in R. 1st part is not a problem, but I'm stuck with the 2nd part: Do you have pointers to where I can learn how to estimate $\mu$ and $\sigma$ in a way that satisfies the 2 conditions (ML and based on counts)? Thank you! – caracal Dec 16 '11 at 13:42
3  
@caracal: as you know, ML methods for the original data are routine and widespread: for the normal distribution, for instance, the MLE of $\mu$ is the sample mean and the MLE of $\sigma$ is the square root of the sample standard deviation (without the usual bias correction). To obtain estimates based on counts, I computed the likelihood function for the counts--this requires computing values of the CDF at the cutpoints, taking their logs, multiplying by the counts, and adding up--and optimized it using generic optimization software. – whuber Dec 16 '11 at 14:35
4  
@caracal You probably no longer need it, but an example of R code for ML fitting of binned data now appears in a related question: stats.stackexchange.com/a/34894. – whuber Aug 22 '12 at 21:29
1  
@Clarinetist The principal point of my answer is to suggest that what you have been taught is based on a confusion of two concepts of DF. Although that confusion causes no problems for standard least-squares Normal-theory models, it leads to errors even in simple, common circumstances like analyses of contingency tables. That matrix rank gives the functional DF. In a least-squares linear model it happens to give the correct DF for certain kinds of tests, such as F tests. For the chi-squared test, the special conditions are enumerated later in the answer as points (1) and (2). – whuber 2 days ago

Or simply: the number of elements in a numerical array that you're allowed to change so that the value of the statistic remains unchanged.

# for instance if:
x + y + z = 10

you can change, for instance, x and y at random, but you cannot change z (you can, but not at random, therefore you're not free to change it - see Harvey's comment), 'cause you'll change the value of the statistic (Σ = 10). So, in this case df = 2.

share|improve this answer
13  
It is not quite correct to say "you cannot change z". In fact, you have to change z to make the sum equal 10. But you have no choice (no freedom) about what it changes to. You can change any two values, but not the third. – Harvey Motulsky Jul 28 '10 at 14:29
1  
That's right, thanks for spotting an error! – aL3xa Jul 28 '10 at 17:35
    
+1 I think this is the nicest/simplest example here. – ars Jul 31 '10 at 5:18

The concept is not at all difficult to make mathematical precise given a bit of general knowledge of $n$-dimensional Euclidean geometry, subspaces and orthogonal projections.

If $P$ is an orthogonal projection from $\mathbb{R}^n$ to a $p$-dimensional subspace $L$ and $x$ is an arbitrary $n$-vector then $Px$ is in $L$, $x - Px$ and $Px$ are orthogonal and $x - Px \in L^{\perp}$ is in the orthogonal complement of $L$. The dimension of this orthogonal complement, $L^{\perp}$, is $n-p$. If $x$ is free to vary in an $n$-dimensional space then $x - Px$ is free to vary in an $n-p$ dimensional space. For this reason we say that $x - Px$ has $n-p$ degrees of freedom.

These considerations are important to statistics because if $X$ is an $n$-dimensional random vector and $L$ is a model of its mean, that is, the mean vector $E(X)$ is in $L$, then we call $X-PX$ the vector of residuals, and we use the residuals to estimate the variance. The vector of residuals has $n-p$ degrees of freedom, that is, it is constrained to a subspace of dimension $n-p$.

If the coordinates of $X$ are independent and normally distributed with the same variance $\sigma^2$ then

  • The vectors $PX$ and $X - PX$ are independent.
  • If $E(X) \in L$ the distribution of the squared norm of the vector of residuals $||X - PX||^2$ is a $\chi^2$-distribution with scale parameter $\sigma^2$ and another parameter that happens to be the degrees of freedom $n-p$.

The sketch of proof of these facts is given below. The two results are central for the further development of the statistical theory based on the normal distribution. Note also that this is why the $\chi^2$-distribution has the parametrization it has. It is also a $\Gamma$-distribution with scale parameter $2\sigma^2$ and shape parameter $(n-p)/2$, but in the context above it is natural to parametrize in terms of the degrees of freedom.

I must admit that I don't find any of the paragraphs cited from the Wikipedia article particularly enlightening, but they are not really wrong or contradictory either. They say in an imprecise, and in a general loose sense, that when we compute the estimate of the variance parameter, but do so based on residuals, we base the computation on a vector that is only free to vary in a space of dimension $n-p$.

Beyond the theory of linear normal models the use of the concept of degrees of freedom can be confusing. It is, for instance, used in the parametrization of the $\chi^2$-distribution whether or not there is a reference to anything that could have any degrees of freedom. When we consider statistical analysis of categorical data there can be some confusion about whether the "independent pieces" should be counted before or after a tabulation. Furthermore, for constraints, even for normal models, that are not subspace constraints, it is not obvious how to extend the concept of degrees of freedom. Various suggestions exist typically under the name of effective degrees of freedom.

Before any other usages and meanings of degrees of freedom is considered I will strongly recommend to become confident with it in the context of linear normal models. A reference dealing with this model class is A First Course in Linear Model Theory, and there are additional references in the preface of the book to other classical books on linear models.

Proof of the results above: Let $\xi = E(X)$, note that the variance matrix is $\sigma^2 I$ and choose an orthonormal basis $z_1, \ldots, z_p$ of $L$ and an orthonormal basis $z_{p+1}, \ldots, z_n$ of $L^{\perp}$. Then $z_1, \ldots, z_n$ is an orthonormal basis of $\mathbb{R}^n$. Let $\tilde{X}$ denote the $n$-vector of the coefficients of $X$ in this basis, that is $$\tilde{X}_i = z_i^T X.$$ This can also be written as $\tilde{X} = Z^T X$ where $Z$ is the orthogonal matrix with the $z_i$'s in the columns. Then we have to use that $\tilde{X}$ has a normal distribution with mean $Z^T \xi$ and, because $Z$ is orthogonal, variance matrix $\sigma^2 I$. This follows from general linear transformation results of the normal distribution. The basis was chosen so that the coefficients of $PX$ are $\tilde{X}_i$ for $i= 1, \ldots, p$, and the coefficients of $X - PX$ are $\tilde{X}_i$ for $i= p+1, \ldots, n$. Since the coefficients are uncorrelated and jointly normal, they are independent, and this implies that $$PX = \sum_{i=1}^p \tilde{X}_i z_i$$ and $$X - PX = \sum_{i=p+1}^n \tilde{X}_i z_i$$ are independent. Moreover, $$||X - PX||^2 = \sum_{i=p+1}^n \tilde{X}_i^2.$$ If $\xi \in L$ then $E(\tilde{X}_i) = z_i^T \xi = 0$ for $i = p +1, \ldots, n$ because then $z_i \in L^{\perp}$ and hence $z_i \perp \xi$. In this case $||X - PX||^2$ is the sum of $n-p$ independent $N(0, \sigma^2)$-distributed random variables, whose distribution, by definition, is a $\chi^2$-distribution with scale parameter $\sigma^2$ and $n-p$ degrees of freedom.

share|improve this answer
    
NRH, Thanks! (1) Why is $E(X)$ required to be inside $L$? (2) Why $PX$ and $X−PX$ are independent? (3) Is the dof in the random variable context defined from the dof in its deterministic case? For example, is the reason for $||X−PX||^2$ has dof $n-p$ because it is true when $X$ is a deterministic variable instead of a random variable? (4) Are there references (books, papers or links) that hold the same/similar opinion as yours? – Tim Oct 12 '11 at 22:50
    
@Tim, $PX$ and $X-PX$ are independent, since they are normal and uncorrelated. – mpiktas Oct 13 '11 at 7:34
    
@Tim, I have reworded the answer a little and given a proof of the stated results. The mean is required to be in $L$ to prove the result about the $\chi^2$-distribution. It is a model assumption. In the literature you should look for linear normal models or general linear models, but right now I can only recall some old, unpublished lecture notes. I will see if I can find a suitable reference. – NRH Oct 13 '11 at 11:06
    
Wonderful answer. Thanks for the insight. One question: I got lost what you meant by the phrase "the mean vector $EX$ is in $L$". Can you explain? Are you try to define $E$? to define $L$? something else? Maybe this sentence is trying to do too much or be too concise for me. Can you elaborate what is the definition of $E$ in the context you mention: is it just $E(x_1,x_2,\dots,x_n) = (x_1+x_2+\dots+x_n)/n$? Can you elaborate on what is $L$ in this context (of normal iid coordinates)? Is it just $L = \mathbb{R}$? – D.W. Oct 13 '11 at 21:12
    
@D.W. The $E$ is the expectation operator. So $E(X)$ is the vector of coordinatewise expectations of $X$. The subspace $L$ is any $p$-dimensional subspace of $\mathbb{R}^n$. It is a space of $n$-vectors and certainly not $\mathbb{R}$, but it can very well be one-dimensional. The simplest example is perhaps when it is spanned by the $\mathbf{1}$-vector with a 1 at all $n$-coordinates. This is the model of all coordinates of $X$ having the same mean value, but many more complicated models are possible. – NRH Oct 13 '11 at 22:02

It's really no different from the way the term "degrees of freedom" works in any other field. For example, suppose you have four variables: the length, the width, the area, and the perimeter of a rectangle. Do you really know four things? No, because there are only two degrees of freedom. If you know the length and the width, you can derive the area and the perimeter. If you know the length and the area, you can derive the width and the perimeter. If you know the area and the perimeter you can derive the length and the width (up to rotation). If you have all four, you can either say that the system is consistent (all of the variables agree with each other), or inconsistent (no rectangle could actually satisfy all of the conditions). A square is a rectangle with a degree of freedom removed; if you know any side of a square or its perimeter or its area, you can derive all of the others because there's only one degree of freedom.

In statistics, things get more fuzzy, but the idea is still the same. If all of the data that you're using as the input for a function are independent variables, then you have as many degrees of freedom as you have inputs. But if they have dependence in some way, such that if you had n - k inputs you could figure out the remaining k, then you've actually only got n - k degrees of freedom. And sometimes you need to take that into account, lest you convince yourself that the data are more reliable or have more predictive power than they really do, by counting more data points than you really have independent bits of data.

(Taken from a post at http://www.reddit.com/r/math/comments/9qbut/could_someone_explain_to_me_what_degrees_of/c0dxtbq?context=3.)

Moreover, all three definitions are almost trying to give a same message.

share|improve this answer
    
Basically right, but I'm concerned that the middle paragraph could be read in a way that confuses correlation, independence (of random variables), and functional independence (of a manifold of parameters). The correlation-independence distinction is particularly important to maintain. – whuber Oct 12 '11 at 20:46
    
@whuber: is it fine now? – Biostat Oct 12 '11 at 20:55
2  
It's correct, but the way it uses terms would likely confuse some people. It still does not explicitly distinguish dependence of random variables from functional dependence. For example, the two variables in a (nondegenerate) bivariate normal distribution with nonzero correlation will be dependent (as random variables) but they still offer two degrees of freedom. – whuber Oct 12 '11 at 21:02
5  
This was copy-pasted from a reddit post I made in 2009. – hobbs Apr 20 '14 at 3:22
1  
Our Help Center provide clear guidance on how to reference material written by others, so I hope the OP will come back to this post to take appropriate actions and engage in constructive interactions (we haven't seen him for a while, though). – chl Apr 24 '14 at 18:54

I really like first sentence from The Little Handbook of Statistical Practice. Degrees of Freedom Chapter

One of the questions an instrutor dreads most from a mathematically unsophisticated audience is, "What exactly is degrees of freedom?"

I think you can get really good understanding about degrees of freedom from reading this chapter.

share|improve this answer

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

    
An excellent link. Thanks for sharing this one! – aL3xa Jul 28 '10 at 13:43
5  
It would be nice to have an explanation for why degrees of freedom is important, rather than just what it is. For instance, showing that the estimate of variance with 1/n is biased but using 1/(n-1) yields an unbiased estimator. – Tristan Jul 31 '10 at 20:12

Wikipedia asserts that degrees of freedom of a random vector can be interpreted as the dimensions of vector subspaces. I want to go step-by-step, very basically through this as a partial answer and elaboration on the Wikipedia entry.

The example proposed is that of a random vector corresponding to the measurements of a continuous variable for different subjects, expressed as a vector extending from the origin $[a\,b\,c]^T$. Its orthogonal projection on the vector $ [1\,1\,1]^T $ results in a vector equal to the multiplication of the mean of the measurements $\bar{x}=1/3(a+b+c)$ times the vector of 1's, $[1\,1\,1]^T $ This projection onto the subspace spanned by the vector of ones has $1\, degree\, of\, freedom$. The residual vector (distance from the mean) is the least-squares projection onto the $(n − 1)$-dimensional orthogonal complement of this subspace, and has $n − 1\, degrees\, of\, freedom$, $n$ being the total number of components of the vector (in our case $3$ since we are in $\mathbb{R}^3$ in the example).This can be simply proven by obtaining the dot product of $[\bar{x}\,\bar{x}\,\bar{x}]^T$ with the difference between $[a\,b\,c]^T$ and $[\bar{x}\,\bar{x}\,\bar{x}]^T$:

$ [\bar{x}\, \bar{x}\,\bar{x}]^T\, \begin{bmatrix} a-\bar{x}\\b-\bar{x}\\c-\bar{x}\end{bmatrix}$

$= \bigg[\frac{(a+b+c)}{3}\, \bigg(a-\frac{(a+b+c)}{3}\bigg)\bigg]+ \bigg[\frac{(a+b+c)}{3} \,\bigg(b-\frac{(a+b+c)}{3}\bigg)\bigg]+ \bigg[\frac{(a+b+c)}{3} \,\bigg(c-\frac{(a+b+c)}{3}\bigg)\bigg] =\frac{(a+b+c)}{3}\bigg[ \bigg(a-\frac{(a+b+c)}{3}\bigg)+ \bigg(b-\frac{(a+b+c)}{3}\bigg)+ \bigg(c-\frac{(a+b+c)}{3}\bigg)\bigg]$

$= \frac{(a+b+c)}{3}\bigg[\frac{1}{3} \bigg(3a-(a+b+c)+ 3b-(a+b+c)+3c-(a+b+c)\bigg)\bigg] =\frac{(a+b+c)}{3}\bigg[\frac{1}{3} (3a-3a+ 3b-3b+3c-3c)\bigg]=0$.

And this relationship extends to any point in a plane orthogonal to $[\bar{x}\,\bar{x}\,\bar{x}]^T$ that includes $[\bar{x}\,\bar{x}\,\bar{x}]^T$. This concept is important in understanding why $\frac 1 {\sigma^2} \Big((X_1-\bar X)^2 + \cdots + (X_n - \bar X)^2 \Big) \sim \chi^2_{n-1}$, a step in the derivation of the t-distribution(here and here).

Let's take the point $[35\,50\,80]^T$, corresponding to three observations. The mean is $55$, and the vector $[55\,\,55\,\,55]^T$ is the normal (orthogonal) to a plane, $55x + 55y + 55z = D$. Plugging in the point coordinates into the plane equation, $D = -9075$.

Now we can choose any other point in this plane, and the mean of its coordinates is going to be $55$, geometrically corresponding to its projection onto the vector $[1\,\,1\,\,1]^T$. Hence for every mean value (in our example, $55$) we can choose an infinite number of pairs of coordinates in $\mathbb{R}^2$ without restriction ($2\, degrees\, of\, freedom$); yet, since the plane is in $\mathbb{R}^3$, the third coordinate will come determined by the equation of the plane (or, geometrically the orthogonal projection of the point onto $[55\,\,55\,\,55]^T$.

Here is representation of three points (in white) lying on the plane (cerulean blue) orthogonal to $[55\,\,55\,\,55]^T$ (arrow): $[35\,\,50\,\,80]^T$, $[80\,\,80\,\,5]$ and $[90\,\,15\,\,60]$ all of them on the plane (subspace with $2\, df$), and then with a mean of their components of $55$, and an orthogonal projection to $[1\,\,1\,\,1]^T$ (subspace with $1\,df$) equal to $[55\,\,55\,\,55]^T$:

enter image description here

share|improve this answer

In my classes, I use one "simple" situation that might help you wonder and perhaps develop a gut feeling for what a degree of freedom may mean.

It is kind of a "Forrest Gump" approach to the subject, but it is worth the try.

Consider you have 10 independent observations $X_1, X_2, \ldots, X_{10}\sim N(\mu,\sigma^2)$ that came right from a normal population whose mean $\mu$ and variance $\sigma^2$ are unknown.

Your observations bring to you collectively information both about $\mu$ and $\sigma^2$. After all, your observations tend to be spread around one central value, which ought to be close to the actual and unknown value of $\mu$ and, likewise, if $\mu$ is very high or very low, then you can expect to see your observations gather around a very high or very low value respectively. One good "substitute" for $\mu$ (in the absence of knowledge of its actual value) is $\bar X$, the average of your observation.

Also, if your observations are very close to one another, that is an indication that you can expect that $\sigma^2$ must be small and, likewise, if $\sigma^2$ is very large, then you can expect to see wildly different values for $X_1$ to $X_{10}$.

If you were to bet your week's wage on which should be the actual value of $\mu$, you would need to choose a pair of values in which you would bet your money. Let's not think of anything as dramatic as losing your paycheck unless you guess $\mu$ correctly until its 200th decimal position. Nope. Let's think of some sort of prizing system that the closer you guess $\mu$ the more you get rewarded.

In some sense, your better, more informed, and more polite guess for $\mu$'s value could be $\bar X$. In that sense, you estimate that $\mu$ must be some value around $\bar X$. Similarly, one good "substitute" for $\sigma^2$ (not required for now) is $S^2$, your sample variance, which makes a good estimate for $\sigma$.

If your were to believe that those substitutes are the actual values of $\mu$ and $\sigma 2$, you would probably be wrong, because are very slim the chances that you were so lucky that your observation coordinated themselves to get you the gift of $\bar X$ being equal to $\mu$ and $S^2$ equal to $\sigma^2$. Nah, probably didn't happen.

But you could be at different levels of wrong, varying from a bit wrong to really, really, really miserably wrong (a.k.a., "Bye-bye, paycheck; see you next week!").

Ok, let's say that you took $\bar X$ as your guess for $\mu$. Consider just two scenarios: $S^2=2$ and $S^2=20,000,000$. In the first, your observations sit pretty and close to one another. In the latter, your observations vary wildly. In which scenario you should be more concerned with your potential losses? If you thought of the second one, you're right. Having a estimate about $\sigma^2$ very reasonably changes your confidence on your bet, for the larger $\sigma^2$ is, the wider you can expect $\bar X$ to variate.

But, beyond information about $\mu$ and $\sigma^2$, your observations also carry some degree of just random fluctuation that is not informative neither about $\mu$ nor about $\sigma^2$.

How can you notice it?

Well, let's assume, for sake of argument, that there is a God and that He has spare time enough to give Himself to the frivolity of telling you specifically the real (and so far unknown) values of both $\mu$ and $\sigma$.

And this is the annoying plot twist of this lysergic tale: He tells it to you after you placed your bet. Perhaps to enlighten you, perhaps to prepare you, perhaps to mock you. How could you

Well, that makes the information about $\mu$ and $\sigma^2$ contained in your observations quite useless now. Your observations' central position $\bar X$ and sample variance $S^2$ are no longer of any help to get closer to the actual values of $\mu$ and $\sigma^2$, for you already know them.

One of the benefits of your good acquaintance with God is that you actually know by how much you failed to guess correctly $\mu$ by using $\bar X$, that is, $(\bar X - \mu)$ your estimation error.

Well, since $X_i\sim N(\mu,\sigma^2)$, then $\bar X\sim N(\mu,\sigma^2/10)$ (trust me in that if you will), also $(\bar X - \mu)\sim N(0,\sigma^2/10)$ (ok, trust me in that on too) and, finally, $$ \frac{\bar X - \mu}{\sigma/\sqrt{10}} \sim N(0,1) $$ (guess what? trust me in that one as well), which carries absolutely no information about $\mu$ or $\sigma^2$.

You know what? If you took any of your individual observations as a guess for $\mu$, your estimation error $(X_i-\mu)$ would be distributed as $N(0,\sigma^2)$. Well, between estimating $\mu$ with $\bar X$ and any $X_i$, choosing $\bar X$ would be better business, because $Var(\bar X) = \sigma^2/10 < \sigma^2 = Var(X_i)$, so $\bar X$ was less prone to be astray from $\mu$ than an individual $X_i$.

Anyway, $(X_i-\mu)/\sigma\sim N(0,1)$, which is also absolutely non informative about neither $\mu$ nor $\sigma^2$.

"Will this tale ever end?" you may be thinking. You also may be thinking "Is there any more random fluctuation that is non informative about $\mu$ and $\sigma^2$?".

[I prefer to think that you are thinking of the latter.]

Yes, there is!

The square of your estimation error for $\mu$ with $X_i$ divided by $\sigma$, $$ \frac{(X_i-\mu)^2}{\sigma^2} = \left(\frac{X_i-\mu}{\sigma}\right)^2 \sim \chi^2 $$ has a Chi-squared distribution, which is the distribution of the square $Z^2$ of a standard Normal $Z\sim N(0,1)$, which I am sure you noticed has absolutely no information about either $\mu$ nor $\sigma^2$.

That is a very well known distribution that arises naturally from the very scenario of you gambling problem for every single one of your ten observations and also from your mean: $$ \frac{(\bar X-\mu)^2}{\sigma^2/10} = \left(\frac{\bar X-\mu}{\sigma/\sqrt{10}}\right)^2 = \left(N(0,1)\right)^2 \sim\chi^2 $$ and also from the gathering of your ten observations' variation: $$ \sum_{i=1}^{10} \frac{(X_i-\mu)^2}{\sigma^2/10} =\sum_{i=1}^{10} \left(\frac{X_i-\mu}{\sigma/\sqrt{10}}\right)^2 =\sum_{i=1}^{10} \left(N(0,1)\right)^2 =\sum_{i=1}^{10} \chi^2. $$ Now that guy doesn't have a Chi-squared distribution, because he is the sum of ten of those Chi-squared distributions, all of them independent from one another (because so are $X_1, \ldots, X_{10}$). Each one of those single Chi-squared distribution is one contribution of random variability with roughly the same amount of contribution to the sum.

Not that each contribution is mathematically equal to the other nine, but all of them have the same expected behavior in distribution. In that sense, they are somehow symmetric.

Each one of those Chi-square is one contribution to the variability of that sum.

If you had 100 observations, the sum above would be expected to be bigger just because it have more sources of contibutions.

Each of those "sources of contributions" with the same behavior can be called degree of freedom.

Now take one or two steps back, re-read the previous paragraphs if needed to accommodate the sudden arrival of your quested-for degree of freedom.

Yep, each degree of freedom can be thought of as one unit of variability that is obligatorily expected to occur and that brings nothing to the improvement of guessing of $\mu$.

The thing is, you start to count on the behavior of those 10 equivalent sources of variability. If you had 100 observations, you would have 100 independent equally-behaved sources of strictly random fluctuation to that sum.

That sum of 10 Chi-squares gets called a Chi-squared distributions with 10 degrees of freedom from now on, and written $\chi^2_{10}$. We can describe what to expect from it starting from its probability density function, that can be mathematically derived from the density from that single Chi-squared distribution (from now on called Chi-squared distribution with one degree of freedom and written $\chi^2_1$), that can be mathematically derived from the density of the normal distribution.

"So what?" --- You might be thinking --- "That is of any good only if God took the time to tell me the values of $\mu$ and $\sigma^2$, of all the things He could tell me!"

Indeed, if God Almighty were to busy to tell you the values of $\mu$ and $\sigma^2$, you would still have that 10 sources, that 10 degrees of freedom.

Things start to get weird (Hahahaha; only now!) when you rebel against God and try and get along all by yourself, without expecting Him to patronize you.

You have $\bar X$ and $S^2$, estimators for $\mu$ and $\sigma^2$. You can find your way to a safe bet.

You could consider calculating the sum above with $\bar X$ and $S^2$ in the places of $\mu$ and $\sigma^2$: $$ \sum_{i=1}^{10} \frac{(X_i-\bar X)^2}{S^2/10} =\sum_{i=1}^{10} \left(\frac{X_i-\bar X}{S/\sqrt{10}}\right)^2, $$ but that is not the same as the original sum.

"Why not?" The term inside the square of both sums are very different. For instance, it is unlikely but possible that all your observations end up being larger than $\mu$, in which case $(X_i-\mu) > 0$, which implies $\sum_{i=1}^{10}(X_i-\mu) > 0$, but, by its turn, $\sum_{i=1}^{10}(X_i-\bar X) = 0$, because $\sum_{i=1}^{10}X_i-10 \bar X =10 \bar X - 10 \bar X = 0$.

Worse, you can prove easily (Hahahaha; right!) that $\sum_{i=1}^{10}(X_i-\bar X)^2 \le \sum_{i=1}^{10}(X_i-\mu)^2$ with strict inequality when at least two observations are different (which is not unusual).

"But wait! There's more!" $$ \frac{X_i-\bar X}{S/\sqrt{10}} $$ doesn't have standard normal distribution, $$ \frac{(X_i-\bar X)^2}{S^2/10} $$ doesn't have Chi-squared distribution with one degree of freedom, $$ \sum_{i=1}^{10} \frac{(X_i-\bar X)^2}{S^2/10} $$ doesn't have Chi-squared distribution with 10 degrees of freedom $$ \frac{\bar X-\mu}{S/\sqrt{10}} $$ doesn't have standard normal distribution.

"Was it all for nothing?"

No way. Now comes the magic! Note that $$ \sum_{i=1}^{10} \frac{(X_i-\bar X)^2}{\sigma^2} =\sum_{i=1}^{10} \frac{[X_i-\mu+\mu-\bar X]^2}{\sigma^2} =\sum_{i=1}^{10} \frac{[(X_i-\mu)-(\bar X-\mu)]^2}{\sigma^2} =\sum_{i=1}^{10} \frac{(X_i-\mu)^2-2(X_i-\mu)(\bar X-\mu)+(\bar X-\mu)^2}{\sigma^2} =\sum_{i=1}^{10} \frac{(X_i-\mu)^2-(\bar X-\mu)^2}{\sigma^2} =\sum_{i=1}^{10} \frac{(X_i-\mu)^2}{\sigma^2}-\sum_{i=1}^{10} \frac{(\bar X-\mu)^2}{\sigma^2} =\sum_{i=1}^{10} \frac{(X_i-\mu)^2}{\sigma^2}-10\frac{(\bar X-\mu)^2}{\sigma^2} =\sum_{i=1}^{10} \frac{(X_i-\mu)^2}{\sigma^2}-\frac{(\bar X-\mu)^2}{\sigma^2/10} $$ or, equivalently, $$ \sum_{i=1}^{10} \frac{(X_i-\mu)^2}{\sigma^2} =\sum_{i=1}^{10} \frac{(X_i-\bar X)^2}{\sigma^2} +\frac{(\bar X-\mu)^2}{\sigma^2/10}. $$ Now we get back to known fa(e)ces.

The first term has Chi-squared distribution with 10 degrees of freedom and the last term has Chi-squared distribution with one degree of freedom(!).

We simply split a Chi-square with 10 independent equally-behaved sources of variability in two parts, both positive: one part is a Chi-square with one source of variability and the other we can prove (leap of faith? win by WO?) to be also a Chi-square with 9 (=10-1) independent equally-behaved sources of variability, with both parts independent from one another.

This is already a good news, since now we have its distribution.

Alas, it uses $\sigma^2$, to which we have no access (recall that God is amusing Himself on watching our struggle).

Well, $$ S^2=\frac{1}{10-1}\sum_{i=1}^{10} (X_i-\bar X)^2, $$ so $$ \sum_{i=1}^{10} \frac{(X_i-\bar X)^2}{\sigma^2} =\frac{\sum_{i=1}^{10} (X_i-\bar X)^2}{\sigma^2} =\frac{(10-1)S^2}{\sigma^2} \sim\chi^2_{(10-1)} $$ therefore $$ \frac{\bar X-\mu}{S/\sqrt{10}} =\frac{\frac{\bar X-\mu}{\sigma/\sqrt{10}}}{\frac{S}{\sigma}} =\frac{\frac{\bar X-\mu}{\sigma/\sqrt{10}}}{\sqrt{\frac{S^2}{\sigma^2}}} =\frac{\frac{\bar X-\mu}{\sigma/\sqrt{10}}}{\sqrt{\frac{\frac{(10-1)S^2}{\sigma^2}}{(10-1)}}} =\frac{N(0,1)}{\sqrt{\frac{\chi^2_{(10-1)}}{(10-1)}}}, $$ which is a distribution that is not the standard normal, but whose density can be derived from the densities of the standard normal and the Chi-squared with $(10-1)$ degrees of freedom.

One very, very smart guy did that math in the beginning of 20th century and, as an unintended consequence, he made his boss the absolute world leader in the industry of Stout beer. I am talking about William Sealy Gosset (a.k.a. Student; yes, that Student, from the $t$ distribution) and Saint James's Gate Brewery (a.k.a. Guinness Brewery), of which I am a devout.

That, my dear friend, is the origin of the $t$ distribution with $(10-1)$ degrees of freedom. The ratio of a standard normal and the squared root of an independent Chi-square divided by its degrees of freedom, which, in an unpredictable turn of tides, wind up describing the expected behavior of the estimation error you undergo when using the sample average $\bar X$ to estimate $\mu$ and using $S^2$ to estimate the variability of $\bar X$.

There you go. With an awful lot of technical details grossly swept behind the hug, but not depending solely on God's intervention to dangerously bet your whole paycheck.

Bye! =)

share|improve this answer

protected by whuber Oct 24 '13 at 6:11

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.