# How are degrees of freedom related to the alpha level criterion?

I think I grasped what degrees of freedom are. In Fox (2008) Applied Regression Analysis and the GLM (2nd. edition) chapter ten, on page 225 the geometrical representation of degrees of freedom is explained in a lot of detail:

Degrees of freedom associated with a sum of squares represent the dimension of the subspace to which the corresponding vector is confined.

I am currently reading Using Multivariate Statistics by Barbara G. Tabachnick and Linda S. Fidell. (6th. edition) In chapter 2.6, page 81 they discuss degrees of freedom in the context of Specific Comparisons.

In analysis, degrees of freedom are best thought of as a nonrenewable ressource. They are analyzed once with conventional alpha levels, but further analysis require very stringent alpha levels.

They then lament that researches often "spend" degrees of freedom on uninteresting ANOVAS while planned comparisons are neglected. I added the specific paragraph in question:

So. I thought that I understand what degrees of freedom are. I also thought that I know that you have to limit the number of statistical tests/comparisons in your analysis (or adjust your alpha level) to avoid a rejection of H0 when it is actually "true". However, I fail to see how degrees of freedom and multiple comparison testings are linked.

• In what way do we "spend" a degree of freedom for each test?
• How are they nonrenewable ressources?
• How are degrees of freedom relevant for your comparisons in the way it is suggested by Tabachnick and Fidell?
• Perhaps some of what you are asking is already answered at stats.stackexchange.com/questions/16921/…, which is devoted to explaining degrees of freedom generally. I don't think it's a duplicate, because your question focuses on a subtle generalization of the concept and its connection with testing and model development, but it could supply useful background information for you. – whuber Jun 8 '17 at 22:48
• I'd beware of some of what Tabachnick & Fidell say; to my recollection the book has a number of errors and misunderstandings. It's hard to tell from the quote whether that applies here, though. – Glen_b Jun 9 '17 at 6:53
• @whuber So I have read your excelent post about degrees of freedom. However, I am still quite puzzled how degrees of freedom are "lost" which each comparison. – Braino Jun 10 '17 at 0:43
• @Braino Lisa Harlow's training is in psychology as is the case with both Tabachnick and Fidell. If you're seeking an opinion of the standard of the statistical content of something, it might be good to consider whether anyone who is mainly trained in statistics thought it was good. [That's not to say they could not be good statisticians, which is a separate issue -- it's not at all hard to find people from psychology who are very good statisticians. However, it might be good to consider opinions on statistics more broadly than within the limits of that one non-statistics discipline] – Glen_b Jun 10 '17 at 1:37
• I will say I haven't looked at any of the more recent editions of Tabachnick & Fidell -- perhaps the issues with some of the older editions are no longer present -- but I wouldn't simply accept that it's all fine without actually subjecting it to a properly critical eye. There's been an ongoing lack of proper critical assessment of many such texts across a number of disciplines. – Glen_b Jun 10 '17 at 1:39

They're doing what it sounded like -- multiple testing; I had hoped there was something more that would make it clear what they were getting at (which is why I asked for context).

The obvious cost with multiple testing is in overall type I error rate. It's not clear that degrees of freedom would be used up if they're all planned tests -- at least in the direct sense that there's no additional loss of df in the distributions involved in those tests.

Data-driven sequences of tests that lead to different models would use "researcher degrees of freedom", and in that situation there's definitely an issue with d.f. that impacts the tests and standard errors and so on. I don't think that's the issue here.

There's at least one other sense in which we might talk about using up degrees of freedom, in that as you do more comparisons, they're no longer independent of each other -- once you have $n$ "independent" pieces of information, there aren't any more. For example, if you determine that $\hat{\mu}_2-\hat{\mu}_1=3.7$ and you determine that $\hat{\mu}_3-\hat{\mu}_1=5.2$ then $\hat{\mu}_3-\hat{\mu}_2=5.2-3.7$ ... there's no "room" for that to be anything else. In that particular sense, at least, there's fewer degrees of freedom for things to be different from what you might expect given what you already know. It's possible that's what they're getting at, but this is not necessarily a problem for your inference, only for the number of actually different things you can say about your data.

My cursory reading of this section gives me the impression that there is some critical conflation of "degrees of freedom" with "multiplicity".

These are only similar insofar as ignoring degrees of freedom corrections as well as ignoring multiple comparisons leads to anti-conservative inference: you are more likely to say that something is significant when it is not. That is a weak connection, at best, because there are virtually infinite settings in which anti-conservative inference can be achieved through poor statistical practice, so each principle should be discussed in its own right.

Degrees of freedom are corrections made to the null sampling distributions imposed on inferential statistics for a single comparison. These corrections result from estimating other parameters, but the actual estimate is inconsequential. For instance, a T-test makes a comparison of mean differences between two groups. The actual mean difference is a useful summary measure, however, without knowledge of the standard error, it is contextually useless to know the mean difference. Thus, we base our inference on the sampling distribution of the Wald statistic: $T = \hat{d} / se(\hat{d})$. Only this standardized difference allows us to use the same methods to compare concentrations of lead in ground water sources (hopefully very small) and the molecular weights of stars (hopefully very large) with similar methods. This Wald statistic requires finite sample correction using the degrees of freedom. I should also add that, to the best of my knowledge, degrees of freedom are never "analyzed" but rather apply to specific analyses, as with the T-test example.

Multiplicity has the concept of spending your alpha level. The authors seem to suggest, incorrectly, that one may apply inference at the usual 0.05 level and that subsequent testing can be performed but at a more stringent level. Technically, to preserve the 0.05 family-wise-error-rate, all subsequent testing must be performed at the 0 level if one inferential comparison has already been made at the 0.05 level. Bonferroni provides a mean of using a similar threshold evenly for any $k$ comparisons ($k > 2$), but you need not use the same threshold for all comparisons. In clinical trials, a "first glance" at data may use a very conservative alpha spending since the optimism is low that sufficient information growth has occurred. Subsequent comparisons may be more "targeted" in using a larger proportion of the total alpha level.

Estimation and inference are fundamentally different things. It's utterly important to be careful and considerate about why and when a p-value should be presented, if at all, in the results. Corrections for degrees of freedom ensure that even one p-value is correct. But if more than one hypothesis is presented, adjustment for multiple comparisons is not achieved by any correction to the degrees of freedom, but rather using different significance thresholds so that the family wise error rate or false discovery rate is conserved.

• Thank you very much for your reply! So this simply might be a matter of confusing the concept of correction for the unbiased estimator for a single test with the concept of alpha error cummulation by testing multiple hypothesis at the same alpha level? – Braino Jun 12 '17 at 19:28
• @Braino when degrees of freedom corrections are used for estimators (like the standard error, or residual error from a linear regression model), those are biased estimators which are being corrected. But yes it is essentially correct. – AdamO Jun 12 '17 at 21:10