# Bias-variance tradeoff in the paired t-test

Suppose we have $K$ subjects and a treatment with two levels, "Before" and "After". A paired t-test is equivalent to fitting a fixed effects ANOVA:

$Y = Subject + Treatment +\epsilon$

It is also equivalent Repeated Measures ANOVA or Mixed ANOVA, where Treatment is fixed and Subject is random. It tried all four methods for this simple dataset and the Treatment p-value is exactly the same. For the sake of this question, let us consider the fixed effects version only.

The total number of observations is $2K$, while the number of parameters is:

$p = intercept + (K-1)$ subject effects + (2 - 1) treatment effects = $K + 1$

That is, there are about 2 observations per parameter for any $K$. To me it suggests that the model is likely to be very overfitted, unless there is a very substantial subject effect.

In practice, how often have you seen the subject effect so large that it justifies pairing the observations by subject?

• Note that treatment & subject are orthogonal so it's simply a matter of whether the error variance within subjects is lower enough than that between subjects to offset its fewer degrees of freedom - rules of thumb for a minimum no. observations per regression degree of freedom to avoid over-fitting are made up for observational data where predictors are often correlated. – Scortchi - Reinstate Monica Aug 21 '14 at 15:01
• I don't understand why orthogonality should change how the bias-variance tradeoff principle is applied. You seem to suggest that for an orthogonal design, it's ok to add more factors as long as MSE goes down. But (assuming that MSE is the criterion of choice) the same thing can be said about a non-orthogonal design as well. – James Aug 21 '14 at 15:27
• Sorry, I didn't mean to seem to suggest that. There are two considerations: (1) Orthogonality ensures the estimate of the treatment effect is independent of the subject effects - it's the same whether you include subjects or not, & the only issue's getting a better estimate of its standard error; & (2) including subjects in the model may increase prediction error (though orthogonal designs can typically get away with fewer observations per regression degree of freedom than rules of thumb for observational studies suggest is necessary). The first consideration is the relevant one for people ... – Scortchi - Reinstate Monica Aug 22 '14 at 8:47
• ... interested only in effect estimation & who carry out the t-test or its equivalent ANOVAR to this end. – Scortchi - Reinstate Monica Aug 22 '14 at 8:47
• orthogonal designs can typically get away with fewer observations per regression degree of freedom than rules of thumb for observational studies suggest is necessary - I.e., all other things being equal, a if there are p parameters in a model with an orthogonal design, it will deliver a better fit (explain the outcome better) than a model with p parameters and a non-orthogonal design. Did I get your right? – James Aug 22 '14 at 14:17

It's a question of how the data were collected. In a paired design, you do a separate randomization on each subject, so you need to account for that by including the subject effect in the model. You do that because it's mandatory -- nothing to do with overfitting.

Pairing is a special case of blocking. Blocking is considered to be about the most effective thing you can do to control for nuisance variations -- such as variations among subjects. To quote from the flyleaf of Box, Hunter, and Hunter's Statistics For Experimenters (2nd ed.),

"Block what you can and randomize what you can't" can approximately justify an analysis "as if" standard assumptions were true.

Section 4.2 of that book (if not the whole book) is a good reference on this topic. And almost any example in that section is one where it is worth accounting for blocks in the analysis.

• What exactly do you mean by randomization? Each subject get both treatments in the same order, so what is randomized? – James Sep 4 '14 at 19:21
• OK, well then it is a repeated-measures design. All the more reason to include subjects as a factor. In particular, you still have dependence due to subjects, so it is the wrong model if you leave them out. And there really isn't a choice about whether or not to pair -- impossible to do an independent samples design. Note that one common way to analyze repeated measures is to use the paired responses as a multivariate response. You would then not have Subject in the model but you would have no more degrees of freedom. – rvl Sep 4 '14 at 20:58
• PS -- it seems to me that if you're really concerned about bias, the bigger issue is the fact that the order of the treatments isn't randomized - making it impossible to distinguish between treatment effects and order effects. – rvl Sep 4 '14 at 21:11
• James -- if everyone gets the treatments in the same order, how do you separate any potential order effects from treatment effects? – Glen_b -Reinstate Monica Sep 6 '14 at 2:46
• That's a question for someone else. In the paired t-test they probably assume that there is no carryover effect. – James Sep 8 '14 at 15:02