# Why do degrees of freedom impact statistical power?

I have heard that degrees of freedom are bad for statistical power from statisticians at a conference. They were complaining that the degrees of freedom may be an issue.

Can someone explain to me why the degrees of freedom can impact a statistical test?

I am especially interested in the cases where I have one simple linear IV model say \begin{eqnarray} y &=& X \beta + \epsilon \\ X &=& Z \Pi + V \end{eqnarray} where $Z$ is an n by k instrument matrix and X is an n by p endogenous regressor matrix.

For example, the full-vector hypothesis test of the null that all parameters are zero is the Anderson-Rubin test which has a chi-squared distribution. When moving to the subset test, which tests for each individual parameter in $\beta$, the degrees of freedom will drop. That has an impact on power which is why the original question arose in the first place.

• Can you clarify the context you heard this? In some cases, degrees of freedom can be good for statistical power - in a t-test to compare two means, the degrees of freedom reflect your sample sizes, and a large sample size will give you high degrees of freedom and better statistical power. On the other hand, if I fit a regression model which is very complex (so the model has many degrees of freedom) to limited data, then the standard errors on my regression coefficients become large, power is low, and I would have achieved better power with a more parsimonious (lower df) model. Feb 3 '15 at 10:12
• Note that even in the regression example, residual degrees of freedom (rather than model degrees of freedom) are good news for the power, since they represent a greater sample size. (In fact the df in the t-test example are essentially a special example of residual df.) Feb 3 '15 at 10:15
• Hey Folks who down-voted this. I don't understand why since it has attracted two excellent on point answers that help me and everyone who will read this after me. Plus, I clarified my question so you can reconsider your down-votes. Feb 3 '15 at 14:49

In the Anderson and Rubin Statistic case you compute the test statistic directly from the model without using an estimator: your assumption is that at $\beta=\beta_0$ you have $\mathbb{E}\left( Z^\prime[Y-X\beta]\right)=0$. For each instrument $z_k$ you have a central limit theorem (CLT) $\sqrt{n}(\frac{1}{n}\sum_iz_{k,i}(y_i-x_i^\prime\beta_0))\overset{d}{\to} \mathcal{N}(0,\sigma^2_k)$. Stacking all the moments together you get a vector CLT: $\sqrt{n}(Z^\prime(Y-X\beta_0)/N)\overset{d}{\to} \mathcal{N}(0,V)$ where $V=var(\varepsilon_i Z_i)$. You can transform the normal vector into a $\chi^2$ distribution as follow: $AR(\beta_0)=n(Z^\prime(Y-X\beta_0)/N)^\prime V^{-1} (Z^\prime(Y-X\beta_0)/N) \overset{d}{\to} \chi^2_{K}$. This is the Anderson-Rubin test where $K$ the degrees of freedom is also the number of instruments. The more instruments you have, the larger $K$ and the larger the critical values for the $\chi^2$ distribution will be. Hence it becomes harder to reject when K is large. If your data is homoskedastic, then you can do a susbset test which is dominated by a $\chi^2$ with fewer degrees of freedom but this does not work in the heteroskedastic case if the parameters not in the subset are weakly identified... There is a long litterature on this, I think this summarizes the issue well: http://economics.mit.edu/files/9890

In regression there are two kinds of degrees of freedom. As described well below, the denominator or error or residual d.f. is relevant to regression models that have residuals and residual variance. As shown above, this is strongly a function of the sample size, and the more the merrier. On the other hand, numerator d.f. (terminology comes from what's in the numerator of an $F$-test) is the number of free location/regression parameters. When a hypothesis test is conducted, the numerator d.f. is the number of regression parameters that uniquely define the hypothesis. It is the number of restrictions on the parameters.

Take two cases: a categorical variable having 4 levels and quadratic regression. The former has 3 parameters and the latter 2. Adding more categories or more powers of $x$ adds parameters and d.f. for hypothesis tests. Adding numerator d.f. is a good thing if the model has better fit so that the added parameters explain more variation in $Y$. It can be a bad thing if the added parameters don't explain enough extra $Y$ variation to make up for the fact that the critical value for a test statistic becomes more stringent as the numerator d.f. goes up. This happens in order to account for the extra chances you have to find an association. This is easy to see for $\chi^2$ tests where the critical value of the $\chi^2$ distribution keeps going up as (numerator) d.f. go up.

So what the conference speaker said about power being hurt is correct if you compared a low d.f. test with a higher d.f. test and the two versions of the model explained the same amount of variation in $Y$.

It is very important to know that if you ever gave the model "chances" to find associations, those chances must be encapsulated by having the right d.f. in the statistical test. It is not appropriate to excluded unimpressive terms from the model just to lower the d.f.

• Thank you so much! This is exactly the intuition what I was hoping for since I am indeed working on multiple tests that all have the same null hypothesis and the same model but different degrees of freedom. I will write more in my question to illuminate the context. Feb 3 '15 at 14:36
• " It is not appropriate to excluded unimpressive terms from the model just to lower the d.f." - is there any upper bound to that? E.g. if there were 1000s, 10000s, or, say, millions of d.f, would it be a problem at some point? (provided other criteria are met e.g. no smaller model with similar predictive power) Jun 25 '20 at 11:30

Definition & Overview

Degrees of freedom is the number of values that are free to vary when when the value of some statistic, like $\bar{X}$ or $\hat{\sigma}^2$, is known. In other words, it is the number of values that need to be known in order to know all of the values.

Sometimes, such as in the $t$-distribution, degrees of freedom are a parameter of distribution just as mean and variance are parameters of the normal distribution. The degrees of freedom needs to be known in order to uniquely specify the $t$-distribution you are referring to.

Example with $\bar{X}$

As a very simple example, consider $\bar{X}$ with a sample size of $n$. If you know the values $X_1, X_2,..., X_{n-1}$ then you also know the value of $X_n$. So, we say that $\bar{X}$ has $n-1$ degrees of freedom.

$$X_n = n(\bar{X}) - \sum\limits_{i=1}^{n-1} X_i$$

Example with Simple Linear Regression

As a slightly more difficult example, consider the simple linear model $Y_i =\alpha+ \beta x_i + \epsilon_i$ for $i=1,...,n$. Recall the following two identities in linear regression

$$e_1 + e_2 + ... + e_n = 0$$

$$x_1e_1 + x_2e_2 + ... + x_ne_n = 0$$

where the residuals $e_i = \hat{Y}_i - Y_i$. Notice, if we know $e_1,..., e_{n-2}$ but $e_{n-1}$ and $e_n$ are unknown then the above identities give us two equations with two unknowns. Since the number of equations and the number of unknowns is the same, we can solve for $e_{n-1}$ and $e_n$. So, knowing the $n-2$ values $e_1,..., e_{n-2}$ allows us to know all of the values $e_1,...,e_n$ and so the residuals have $n-2$ degrees of freedom.

This can be easily generalized to a multiple linear regression situation with $p$ variables if one recalls that $X^Te = 0$.

Does this Cause a Problem?

If $p$ is close to $n$ we do not have much freedom in the estimates of our fitted values of $\hat{Y}_i$. This is because if we know $e_1,...,e_p$ we also will know $\hat{Y}_1,...,\hat{Y}_p$.

• Thanks so much! This is a useful example with equations! Feb 3 '15 at 14:38
• @TrynnaDoStat I was reading this and I understand what you are getting at, except for one thing; what is the difference between the equations e1+e2+....+en = 0 and x1e1 + x2e2+...+xnen = 0? are those two equations not saying the exact same thing?
– Ryan
Mar 27 '20 at 21:41