2
$\begingroup$

I have heard that degrees of freedom are bad for statistical power from statisticians at a conference. They were complaining that the degrees of freedom may be an issue.

Can someone explain to me why the degrees of freedom can impact a statistical test?

I am especially interested in the cases where I have one simple linear IV model say \begin{eqnarray} y &=& X \beta + \epsilon \\ X &=& Z \Pi + V \end{eqnarray} where $Z$ is an n by k instrument matrix and X is an n by p endogenous regressor matrix.

For example, the full-vector hypothesis test of the null that all parameters are zero is the Anderson-Rubin test which has a chi-squared distribution. When moving to the subset test, which tests for each individual parameter in $\beta$, the degrees of freedom will drop. That has an impact on power which is why the original question arose in the first place.

$\endgroup$
  • 4
    $\begingroup$ Can you clarify the context you heard this? In some cases, degrees of freedom can be good for statistical power - in a t-test to compare two means, the degrees of freedom reflect your sample sizes, and a large sample size will give you high degrees of freedom and better statistical power. On the other hand, if I fit a regression model which is very complex (so the model has many degrees of freedom) to limited data, then the standard errors on my regression coefficients become large, power is low, and I would have achieved better power with a more parsimonious (lower df) model. $\endgroup$ – Silverfish Feb 3 '15 at 10:12
  • 2
    $\begingroup$ Note that even in the regression example, residual degrees of freedom (rather than model degrees of freedom) are good news for the power, since they represent a greater sample size. (In fact the df in the t-test example are essentially a special example of residual df.) $\endgroup$ – Silverfish Feb 3 '15 at 10:15
  • $\begingroup$ Hey Folks who down-voted this. I don't understand why since it has attracted two excellent on point answers that help me and everyone who will read this after me. Plus, I clarified my question so you can reconsider your down-votes. $\endgroup$ – Hirek Feb 3 '15 at 14:49
1
$\begingroup$

In the Anderson and Rubin Statistic case you compute the test statistic directly from the model without using an estimator: your assumption is that at $\beta=\beta_0$ you have $\mathbb{E}\left( Z^\prime[Y-X\beta]\right)=0$. For each instrument $z_k$ you have a central limit theorem (CLT) $\sqrt{n}(\frac{1}{n}\sum_iz_{k,i}(y_i-x_i^\prime\beta_0))\overset{d}{\to} \mathcal{N}(0,\sigma^2_k)$. Stacking all the moments together you get a vector CLT: $\sqrt{n}(Z^\prime(Y-X\beta_0)/N)\overset{d}{\to} \mathcal{N}(0,V)$ where $V=var(\varepsilon_i Z_i)$. You can transform the normal vector into a $\chi^2$ distribution as follow: $AR(\beta_0)=n(Z^\prime(Y-X\beta_0)/N)^\prime V^{-1} (Z^\prime(Y-X\beta_0)/N) \overset{d}{\to} \chi^2_{K}$. This is the Anderson-Rubin test where $K$ the degrees of freedom is also the number of instruments. The more instruments you have, the larger $K$ and the larger the critical values for the $\chi^2$ distribution will be. Hence it becomes harder to reject when K is large. If your data is homoskedastic, then you can do a susbset test which is dominated by a $\chi^2$ with fewer degrees of freedom but this does not work in the heteroskedastic case if the parameters not in the subset are weakly identified... There is a long litterature on this, I think this summarizes the issue well: http://economics.mit.edu/files/9890

$\endgroup$
8
$\begingroup$

In regression there are two kinds of degrees of freedom. As described well below, the denominator or error or residual d.f. is relevant to regression models that have residuals and residual variance. As shown above, this is strongly a function of the sample size, and the more the merrier. On the other hand, numerator d.f. (terminology comes from what's in the numerator of an $F$-test) is the number of free location/regression parameters. When a hypothesis test is conducted, the numerator d.f. is the number of regression parameters that uniquely define the hypothesis. It is the number of restrictions on the parameters.

Take two cases: a categorical variable having 4 levels and quadratic regression. The former has 3 parameters and the latter 2. Adding more categories or more powers of $x$ adds parameters and d.f. for hypothesis tests. Adding numerator d.f. is a good thing if the model has better fit so that the added parameters explain more variation in $Y$. It can be a bad thing if the added parameters don't explain enough extra $Y$ variation to make up for the fact that the critical value for a test statistic becomes more stringent as the numerator d.f. goes up. This happens in order to account for the extra chances you have to find an association. This is easy to see for $\chi^2$ tests where the critical value of the $\chi^2$ distribution keeps going up as (numerator) d.f. go up.

So what the conference speaker said about power being hurt is correct if you compared a low d.f. test with a higher d.f. test and the two versions of the model explained the same amount of variation in $Y$.

It is very important to know that if you ever gave the model "chances" to find associations, those chances must be encapsulated by having the right d.f. in the statistical test. It is not appropriate to excluded unimpressive terms from the model just to lower the d.f.

$\endgroup$
  • $\begingroup$ Thank you so much! This is exactly the intuition what I was hoping for since I am indeed working on multiple tests that all have the same null hypothesis and the same model but different degrees of freedom. I will write more in my question to illuminate the context. $\endgroup$ – Hirek Feb 3 '15 at 14:36
3
$\begingroup$

Definition & Overview

Degrees of freedom is the number of values that are free to vary when when the value of some statistic, like $\bar{X}$ or $\hat{\sigma}^2$, is known. In other words, it is the number of values that need to be known in order to know all of the values.

Sometimes, such as in the $t$-distribution, degrees of freedom are a parameter of distribution just as mean and variance are parameters of the normal distribution. The degrees of freedom needs to be known in order to uniquely specify the $t$-distribution you are referring to.

Example with $\bar{X}$

As a very simple example, consider $\bar{X}$ with a sample size of $n$. If you know the values $X_1, X_2,..., X_{n-1}$ then you also know the value of $X_n$. So, we say that $\bar{X}$ has $n-1$ degrees of freedom.

$$X_n = n(\bar{X}) - \sum\limits_{i=1}^{n-1} X_i$$

Example with Simple Linear Regression

As a slightly more difficult example, consider the simple linear model $Y_i =\alpha+ \beta x_i + \epsilon_i$ for $i=1,...,n$. Recall the following two identities in linear regression

$$e_1 + e_2 + ... + e_n = 0$$

$$x_1e_1 + x_2e_2 + ... + x_ne_n = 0$$

where the residuals $e_i = \hat{Y}_i - Y_i$. Notice, if we know $e_1,..., e_{n-2}$ but $e_{n-1}$ and $e_n$ are unknown then the above identities give us two equations with two unknowns. Since the number of equations and the number of unknowns is the same, we can solve for $e_{n-1}$ and $e_n$. So, knowing the $n-2$ values $e_1,..., e_{n-2}$ allows us to know all of the values $e_1,...,e_n$ and so the residuals have $n-2$ degrees of freedom.

This can be easily generalized to a multiple linear regression situation with $p$ variables if one recalls that $X^Te = 0$.

Does this Cause a Problem?

If $p$ is close to $n$ we do not have much freedom in the estimates of our fitted values of $\hat{Y}_i$. This is because if we know $e_1,...,e_p$ we also will know $\hat{Y}_1,...,\hat{Y}_p$.

$\endgroup$
  • $\begingroup$ Thanks so much! This is a useful example with equations! $\endgroup$ – Hirek Feb 3 '15 at 14:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.