Why do degrees of freedom impact statistical power? I have heard that degrees of freedom are bad for statistical power from statisticians at a conference. They were complaining that the degrees of freedom may be an issue.
Can someone explain to me why the degrees of freedom can impact a statistical test?
I am especially interested in the cases where I have one simple linear IV model say
\begin{eqnarray}
y &=& X \beta + \epsilon \\
X &=& Z \Pi + V
\end{eqnarray}
where $Z$ is an n by k instrument matrix and X is an n by p endogenous regressor matrix.
For example, the full-vector hypothesis test of the null that all parameters are zero is the Anderson-Rubin test which has a chi-squared distribution. When moving to the subset test, which tests for each individual parameter in $\beta$, the degrees of freedom will drop. That has an impact on power which is why the original question arose in the first place.
 A: In regression there are two kinds of degrees of freedom.  As described well below, the denominator or error or residual d.f. is relevant to regression models that have residuals and residual variance.  As shown above, this is strongly a function of the sample size, and the more the merrier.  On the other hand, numerator d.f. (terminology comes from what's in the numerator of an $F$-test) is the number of free location/regression parameters.  When a hypothesis test is conducted, the numerator d.f. is the number of regression parameters that uniquely define the hypothesis. It is the number of restrictions on the parameters.
Take two cases: a categorical variable having 4 levels and quadratic regression.  The former has 3 parameters and the latter 2.  Adding more categories or more powers of $x$ adds parameters and d.f. for hypothesis tests.  Adding numerator d.f. is a good thing if the model has better fit so that the added parameters explain more variation in $Y$.  It can be a bad thing if the added parameters don't explain enough extra $Y$ variation to make up for the fact that the critical value for a test statistic becomes more stringent as the numerator d.f. goes up.  This happens in order to account for the extra chances you have to find an association.  This is easy to see for $\chi^2$ tests where the critical value of the $\chi^2$ distribution keeps going up as (numerator) d.f. go up.
So what the conference speaker said about power being hurt is correct if you compared a low d.f. test with a higher d.f. test and the two versions of the model explained the same amount of variation in $Y$.
It is very important to know that if you ever gave the model "chances" to find associations, those chances must be encapsulated by having the right d.f. in the statistical test.  It is not appropriate to excluded unimpressive terms from the model just to lower the d.f.
A: Definition & Overview
Degrees of freedom is the number of values that are free to vary when when the  value of some statistic, like $\bar{X}$ or $\hat{\sigma}^2$, is known. In other words, it is the number of values that need to be known in order to know all of the values.
Sometimes, such as in  the $t$-distribution, degrees of freedom are a parameter of distribution just as mean and variance are parameters of the normal distribution. The degrees of freedom needs to be known in order to uniquely specify the $t$-distribution you are referring to.
Example with $\bar{X}$
As a very simple example, consider $\bar{X}$ with a sample size of $n$. If you know the values $X_1, X_2,..., X_{n-1}$ then you also know the value of $X_n$. So, we say that $\bar{X}$ has $n-1$ degrees of freedom.
$$X_n = n(\bar{X}) - \sum\limits_{i=1}^{n-1} X_i$$
Example with Simple Linear Regression
As a slightly more difficult example, consider the simple linear model $Y_i =\alpha+ \beta x_i + \epsilon_i$ for $i=1,...,n$. Recall the following two identities in linear regression
$$e_1 + e_2 + ... + e_n = 0$$
$$x_1e_1 + x_2e_2 + ... + x_ne_n = 0$$
where the residuals $e_i = \hat{Y}_i - Y_i$. Notice, if we know $e_1,..., e_{n-2}$ but $e_{n-1}$ and $e_n$ are unknown then the above identities give us two equations with two unknowns. Since the number of equations and the number of unknowns is the same, we can solve for $e_{n-1}$ and $e_n$. So, knowing the $n-2$ values  $e_1,..., e_{n-2}$ allows us to know all of the values $e_1,...,e_n$ and so the residuals have $n-2$ degrees of freedom.
This can be easily generalized to a multiple linear regression situation with $p$ variables if one recalls that $X^Te = 0$.
Does this Cause a Problem?
If $p$ is close to $n$ we do not have much freedom in the estimates of our fitted values of $\hat{Y}_i$. This is because if we know $e_1,...,e_p$ we also will know  $\hat{Y}_1,...,\hat{Y}_p$. 
A: In the Anderson and Rubin Statistic case you compute the test statistic directly from the model without using an estimator: your assumption is that at $\beta=\beta_0$ you have $\mathbb{E}\left( Z^\prime[Y-X\beta]\right)=0$. For each instrument $z_k$ you have a central limit theorem (CLT) $\sqrt{n}(\frac{1}{n}\sum_iz_{k,i}(y_i-x_i^\prime\beta_0))\overset{d}{\to} \mathcal{N}(0,\sigma^2_k)$. Stacking all the moments together you get a vector CLT: $\sqrt{n}(Z^\prime(Y-X\beta_0)/N)\overset{d}{\to} \mathcal{N}(0,V)$ where $V=var(\varepsilon_i Z_i)$. You can transform the normal vector into a $\chi^2$ distribution as follow:
$AR(\beta_0)=n(Z^\prime(Y-X\beta_0)/N)^\prime V^{-1} (Z^\prime(Y-X\beta_0)/N) \overset{d}{\to} \chi^2_{K}$.
This is the Anderson-Rubin test where $K$ the degrees of freedom is also the number of instruments. The more instruments you have, the larger $K$ and the larger the critical values for the $\chi^2$ distribution will be. Hence it becomes harder to reject when K is large. If your data is homoskedastic, then you can do a susbset test which is dominated by a $\chi^2$ with fewer degrees of freedom but this does not work in the heteroskedastic case if the parameters not in the subset are weakly identified... There is a long litterature on this, I think this summarizes the issue well: http://economics.mit.edu/files/9890
