Paired versus unpaired t-test

Question

Suppose I have 20 mice. I pair the mice in some way, so that I get 10 pairs. For the purpose of this question, it could be a random pairing, OR it could be a sensible pairing, like trying to pair mice from the same litter, of the same sex, with similar weight, OR it could be a deliberately stupid pairing like trying to pair mice with weights as unequal as they could possibly be. I then use random numbers to assign one mouse in each pair to the control group and the other mouse to the to-be-treated group. I now do the experiment, treating only the to-be-treated mice, but otherwise paying no attention whatsoever to the arrangements just made.

When one comes to analyze the results, one could either use unpaired t-testing or paired t-testing. In what way, if any, will the answers differ? (I'm basically interested in systematic differences of any statistical parameter that needs to be estimated.)

The reason I ask this is that a paper I was recently involved with was criticized by a biologist for using a paired t-test rather than an unpaired t-test. Of course, in the actual experiment, the situation was not as extreme as the situation I've sketched, and there were, in my opinion, good reasons for pairing. But the biologist didn't agree.

It seems to me that it's not possible to incorrectly improve statistical significance (decrease p-value), in the circumstances I sketched, by using a paired t-test, rather than an unpaired test, even if it is inappropriate to pair. It could however worsen statistical significance if mice were badly paired. Is this right?

Answer 1

I agree with the points that both Frank and Peter make but I think there is a simple formula that gets to the heart of the issue and may be worthwhile for the OP to consider.

Let $X$ and $Y$ be two random variables whose correlation is unknown.

Let $Z=X-Y$

What is the variance of $Z$?

Here is the simple formula: $$ \text{Var}(Z)=\text{Var}(X) + \text{Var}(Y) - 2 \text{Cov}(X,Y). $$ What if $\text{Cov}(X,Y)>0$ (i.e., $X$ and $Y$ are positively correlated)?

Then $\text{Var}(Z)\lt \text{Var}(X)+\text{Var}(Y)$. In this case if the pairing is made because of positive correlation such as when you are dealing with the same subject before and after intervention pairing helps because the independent paired difference has lower variance than the variance you get for the unpaired case. The method reduced variance. The test is more powerful. This can be dramatically shown with cyclic data. I saw an example in a book where they wanted to see if the temperature in Washington DC is higher than in New York City. So they took average monthly temperature in both cities for say 2 years. Of course there is a huge difference over the course of the year because of the four seasons. This variation is too large for an unpaired t test to detect a difference. However pairing based on the same month in the same year eliminates this seasonal effect and the paired $t$-test clearly showed that the average temperature in DC tended to be higher than in New York. $X_i$ (temperature at NY in month $A$) and $Y_i$ (temperature in DC in month $A$) are positively correlated because the seasons are the same in NY and DC and the cities are close enough that they will often experience the same weather systems that affect temperature. DC may be a little warmer because it is further south.

Note that the large the covariance or correlation the greater is the reduction in variance.

Now suppose $\text{Cov}(X,Y)$ is negative.

Then $\text{Var}(Z) \gt \text{Var}(X)+\text{Var}(Y)$. Now pairing will be worse than not pairing because the variance is actually increased!

When $X$ and $Y$ are uncorrelated then it probably doesn't matter which method you use. Peter's random pairing case is like this situation.

Answer 2

Rather than pairing it is probably better to understand the underlying data model. If pairing is done to deal with uncontrolled heterogeneity, it is usually the case (except in twin studies) that the pairing only partially controls this source of variability and multiple regression would do better. This is because matching on continuous variables frequently results in residual variability because of not being able to do exact matching on such variables.

Answer 3

The two tests (paired and unpaired) ask different questions so they can get different answers. Correct pairing nearly always is more powerful than unpaired - that's really the point of pairing. So, since you say the pairing is correct, it is likely that the p-value for your paired test is lower than for the same data unpaired. You could, of course, do both and see for yourself.

Therefore, the answer to your dilemma is substantive, not statistical. Is your pairing right?

Could you get a more significant result from random pairing than from a unpaired test? Let's see:

set.seed(2910110192)
x <- rnorm(100, 10, 2)
y <- rnorm(100, 10, 2)
t.test(x, y)
t.test(x, y, paired = T)

Yes you can, although here the difference is very small, the paired had a lower p. I ran that code several times. Not surprisingly, sometimes one p is lower, sometimes the other, but the difference was small in all cases. However, I am sure that in some situations the difference in p values could be large.

Answer 4

I now understand much better what was worrying me about paired versus unpaired t-tests, and associated p-values. Finding out has been an interesting journey, and there have been many surprises along the way. One surprise has resulted from an investigation of Michael's contribution. This is irreproachable in terms of practical advice. Moreover, he says what I think virtually all statisticians believe, and he has several upvotes to back this up. However, as a piece of theory, it is not literally correct. I discovered this by working out the formulas for the p-values, and then thinking carefully how to use the formulas to lead to counter-examples. I'm a mathematician by training, and the counter-example is a "mathematician's counter-example". It's not something you would come across in practical statistics, but it was the kind of thing I was trying to find out about when I asked my original question.

Here is the R-code that gives the counter-example:

vLength <- 10; meanDiff <-10^9; numSamples <- 3;
pv <- function(vLength,meanDiff) {
    X <- rnorm(vLength)
    Y <- X - meanDiff + rnorm(vLength,sd=0.0001)
    Paired <- t.test(X,Y,var.equal=T,paired=T)
    NotPaired <- t.test(X,Y,var.equal=T,paired=F)
    c(Paired$p.value,NotPaired$p.value,cov(X,Y))
}
ans <- replicate(numSamples,pv(vLength,meanDiff))

Note the following features: X and Y are two 10-tuples whose difference is huge and very nearly constant. To many significant figures, the correlation is 1.000.... The p-value for the unpaired test is around 10^40 times smaller than the p-value for the paired test. So this contradicts Michael's account, provided that one reads his account literally, mathematician-style. Here ends the part of my answer related to Michael's answer.

---

Here are the thoughts prompted by Peter's answer. During the discussion of my original question, I conjectured in a comment that two particular distributions of p-values that sound different are in fact the same. I can now prove this. What is more important is that the proof reveals the fundamental nature of a p-value, so fundamental that no text (that I've come across) bothers to explain. Maybe all professional statisticians know the secret, but to me, the definition of p-value always seemed strange and artificial. Before giving away the statistician's secret, let me specify the question.

Let $n>1$ and choose randomly and independently two random $n$-tuples from some normal distribution. There are two ways of getting a p-value from this choice. One is to use an unpaired t-test, and the other is to use a paired t-test. My conjecture was that the distribution of p-values that one gets is the same in the two cases. When I first started to think about it, I decided that this conjecture had been foolhardy and was false: the unpaired test is associated to a t-statistic on $2(n-1)$ degrees of freedom, and the paired test to a t-statistic on $n-1$ degrees of freedom. These two distributions are different, so how on earth could the associated distributions of p-values be the same? Only after much further thought did I realize that this obvious dismissal of my conjecture was too facile.

The answer comes from the following considerations. Suppose $f:(0,\infty)\to (0,\infty)$ is a continuous pdf (that is, its integral has value one). A change of coordinates converts the associated distribution into the uniform distribution on $[0,1]$. The formula is $$p=\int_t^\infty f(s)\,ds$$ and this much is explained in many texts. What the texts fail to point out in the context of p-values is that this is exactly the formula that gives the p-value from the t-statistic, when $f$ is the pdf for the t-distribution. (I'm trying to keep the discussion as simple as I can, because it really is simple. A fuller discussion would treat one-sided and two-sided t-tests slightly differently, factors of 2 might arise, and the t-statistic might lie in $(-\infty,\infty)$ instead of in $[0,\infty)$. I omit all that clutter.)

Exactly the same discussion applies when finding the p-value associated with any of the other standard distributions in statistics. Once again, if the data is randomly distributed (this time according to some different distribution), then the resulting p-values will be distributed uniformly in $[0,1]$.

How does this apply to our paired and unpaired t-tests? The point is in the paired t-test, with samples chosen independently and randomly, as in my code above, the value of t does indeed follow a t-distribution (with $n-1$ degrees of freedom). So the p-values that result from replicating the choice of X and Y many times follow the uniform distribution on $[0,1]$. The same is true for the unpaired t-test, though this time the t-distribution has $2(n-1)$ degrees of freedom. Nevertheless, the p-values that result also have a uniform distribution on $[0,1]$, by the general argument I gave above. If Peter's code above is applied to determine p-values, then we get two distinct methods of drawing a random sample from the uniform distribution on $[0,1]$. However the two answers are not independent.

Answer 5

I would offer another perspective. Often, pairing is done do reduce bias. Suppose that you are interested in whether exposure E is a risk factor for a continuous outcome Y. For each E+ subject, you get an age and sex matched subject who is E-. Now, we could do either a paired t-test or an unpaired t-test. I think we should account for matching explicitly and conduct a paired t-test. It is more principled in that it takes the design into account. Whether to take matching into account in the analysis is an issue of the bias-variance tradeoff. Accounting for matching in the analysis provides more protection against bias, but can increase variance. Doing an unpaired t-test may be more efficient, but it would not provide any protection against bias.