I am having trouble finding any web-based resource or (even better) publication which details (or even names) the kind of problems which one avoids by running a paired t-test as opposed to an unpaired test when testing the same population twice.

I am surprised resources like wikipedia also do not detail this. Will an unpaired test overestimate or underestimate my p? Or does this depend on other considerations?

Could you help me out?

  • $\begingroup$ It seems to be explained quite clearly in the first paragraph here $\endgroup$
    – Glen_b
    Nov 17 '13 at 0:18
  • $\begingroup$ that simply explains the usage of the paired t-test and when it is relevant, it does not say what would happen if I wouldn't use it in a case where I should (I know, I was also surprised that this isn't in the first paragraph on wiki). Do you know some reference which does say what kind of misjudgment one is prone to when using an unpaired t-test instead oaf a paired one? $\endgroup$
    – TheChymera
    Nov 17 '13 at 0:26
  • 2
    $\begingroup$ Yes it does. It says "A paired difference test uses additional information about the sample that is not present in an ordinary unpaired testing situation, either to increase the statistical power, or to reduce the effects of confounders." -- "increases power" means it reduces the type II error rate; 'reduces the effect of confounders' would reduce the rate of either one type of error or the other, or maybe even both. $\endgroup$
    – Glen_b
    Nov 17 '13 at 1:01
  • $\begingroup$ May thanks! Do you want to write that as a reply so I can accept it? $\endgroup$
    – TheChymera
    Nov 17 '13 at 1:16
  • $\begingroup$ Uh, okay. I can't right away though; might be some hours. If nobody posts something better before I can do it, I'll convert it into an answer. $\endgroup$
    – Glen_b
    Nov 17 '13 at 1:18

In a paired t-test, each observation is of the form: $Z = X - Y$. That means that the variance of each observation is

$$Var(Z) = Var(X-Y) = Var(X) + Var(Y) -2Cov(X,Y)$$

Now, suppose that X,Y and are positively related (for instance before and after scores on a test). The paired t-test will then reduce the $Var(Z)$ by the $-2Cov(X,Y)$ term.

Thus in the equation for the paired t-test (see here) $$ t= \frac{\bar{X_D} - \mu_d}{\sigma_d /\sqrt{n}}$$

you would get the $\sigma_d$ is smaller and be more likely to reject. If for some reason, the paired scores had a negative covariance, then you would get $\sigma_d$ to be larger and be less likely to reject. Most often (and logically so), paired samples have a positive covariance and thus $Var(Z)$ is reduced. This yields to higher power.

If you look at the wikipedia page for statistical power (here), you will see that for a one sided paired t-test the power (as a function of how large the paired difference is under the alternative -- denoted by $\tau$ is:

$$\pi(\tau)\approx 1-\Phi(1.64-\tau\sqrt{n}/\hat{\sigma}_D). $$

As you can see, for larger values of $\sigma_d$, $\Phi(1.64-\tau\sqrt{n}/\hat{\sigma}_D)$ would decrease and $\pi(\tau)$ would increase.

Thus, if the data is paired, if you don't use the paired test you are losing power and are less likely to reject. However, if the unusual (but perhaps possible) situation where paired data are more variable, then you are more likely to reject and can very well be concluding incorrectly.

It is worth noting that if the data is not paired and you use the paired test, then you also lose power. For instance, if you are using a one-sided t-test on a sample size of $2n$ observations but erroneously concluded that observation $1$ is pared with observation $n+1$, observation $2$ is paired with observation $n+2$, etc..., then in a paired t-test your sample size is $n$. By reducing your sample size, you once again lose power.

Hope that helps!

  • $\begingroup$ Great! So the general formulas of paired and unpaired t-test are equal ($$ t= \frac{\bar{X_D} - \mu_d}{\sigma_d /\sqrt{n}}$$) - only that for a paired test the sd is $$Var(Z) = Var(X) + Var(Y) -2Cov(X,Y)$$ while for an unpaired one it is $$Var(Z) = Var(X) + Var(Y)$$ ? so using an unpaired test where I should use a paired one (assuming pairs correlate positively) would give me a smaller $t$, thus a smaller $p$ and thus inflated significance and increase my risk of a false positive. $\endgroup$
    – TheChymera
    Nov 18 '13 at 15:34
  • $\begingroup$ also, I have found another expression of the paired t-test fomula - are they equivalent? $$t = \frac{\sum\limits_{i=1}^n A_{i}-B_{i}}{\sqrt{\frac{n \sum\limits_{i=1}^n (A_{i}-B_{i})^{2} - \left(\sum\limits_{i=1}^n A_{i}-B_{i}\right)^{2}}{n-1}}}$$ $\endgroup$
    – TheChymera
    Nov 18 '13 at 15:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.