22
$\begingroup$

Suppose I have 20 mice. I pair the mice in some way, so that I get 10 pairs. For the purpose of this question, it could be a random pairing, OR it could be a sensible pairing, like trying to pair mice from the same litter, of the same sex, with similar weight, OR it could be a deliberately stupid pairing like trying to pair mice with weights as unequal as they could possibly be. I then use random numbers to assign one mouse in each pair to the control group and the other mouse to the to-be-treated group. I now do the experiment, treating only the to-be-treated mice, but otherwise paying no attention whatsoever to the arrangements just made.

When one comes to analyze the results, one could either use unpaired t-testing or paired t-testing. In what way, if any, will the answers differ? (I'm basically interested in systematic differences of any statistical parameter that needs to be estimated.)

The reason I ask this is that a paper I was recently involved with was criticized by a biologist for using a paired t-test rather than an unpaired t-test. Of course, in the actual experiment, the situation was not as extreme as the situation I've sketched, and there were, in my opinion, good reasons for pairing. But the biologist didn't agree.

It seems to me that it's not possible to incorrectly improve statistical significance (decrease p-value), in the circumstances I sketched, by using a paired t-test, rather than an unpaired test, even if it is inappropriate to pair. It could however worsen statistical significance if mice were badly paired. Is this right?

$\endgroup$

5 Answers 5

26
$\begingroup$

I agree with the points that both Frank and Peter make but I think there is a simple formula that gets to the heart of the issue and may be worthwhile for the OP to consider.

Let $X$ and $Y$ be two random variables whose correlation is unknown.

Let $Z=X-Y$

What is the variance of $Z$?

Here is the simple formula: $$ \text{Var}(Z)=\text{Var}(X) + \text{Var}(Y) - 2 \text{Cov}(X,Y). $$ What if $\text{Cov}(X,Y)>0$ (i.e., $X$ and $Y$ are positively correlated)?

Then $\text{Var}(Z)\lt \text{Var}(X)+\text{Var}(Y)$. In this case if the pairing is made because of positive correlation such as when you are dealing with the same subject before and after intervention pairing helps because the independent paired difference has lower variance than the variance you get for the unpaired case. The method reduced variance. The test is more powerful. This can be dramatically shown with cyclic data. I saw an example in a book where they wanted to see if the temperature in Washington DC is higher than in New York City. So they took average monthly temperature in both cities for say 2 years. Of course there is a huge difference over the course of the year because of the four seasons. This variation is too large for an unpaired t test to detect a difference. However pairing based on the same month in the same year eliminates this seasonal effect and the paired $t$-test clearly showed that the average temperature in DC tended to be higher than in New York. $X_i$ (temperature at NY in month $A$) and $Y_i$ (temperature in DC in month $A$) are positively correlated because the seasons are the same in NY and DC and the cities are close enough that they will often experience the same weather systems that affect temperature. DC may be a little warmer because it is further south.

Note that the large the covariance or correlation the greater is the reduction in variance.

Now suppose $\text{Cov}(X,Y)$ is negative.

Then $\text{Var}(Z) \gt \text{Var}(X)+\text{Var}(Y)$. Now pairing will be worse than not pairing because the variance is actually increased!

When $X$ and $Y$ are uncorrelated then it probably doesn't matter which method you use. Peter's random pairing case is like this situation.

$\endgroup$
5
  • 3
    $\begingroup$ Michael, because "<" and ">" have special meanings on Web pages, to avoid having large swathes of your text simply disappear from view you it is essential that you use $\TeX$ markup for them in equations (the codes are "\lt" and "\gt" respectively). I marked up the two equations that caused this problem for you. In the future, please read what you post immediately after posting it to make sure people are seeing what you thought they would see, and then feel free to flag your post for moderator attention if there is some problem with the markup. $\endgroup$
    – whuber
    Sep 27, 2012 at 16:28
  • $\begingroup$ @whuber Thank you. I generally do check during and after posting because I find that I mess up equations a lot especially when subscripting. Missing this one is unusual and probably happened because it was a long post and I just carelessly went on to something else that I wanted or needed to do. Sometimes a phone call distracts me and I forget to check. Regarding special symbols that cause text to disappear in a post, I have observed that. I think a simple solution is to make sure you leave a space after the symbol. I think that has worked for me in the past. $\endgroup$ Sep 27, 2012 at 16:38
  • $\begingroup$ +1, really on-point. Note that if $X$ & $Y$ are perfectly uncorrelated in your sample, $\text{Var}(Z)=\text{Var}(X)+\text{Var}(Y)$. $\endgroup$ Aug 23, 2013 at 3:23
  • $\begingroup$ @MichaelChernick For the case when Cov(X,Y)<0, I have a question: If my goal is to infer E[X]-E[Y] from my experiment, then EVEN THOUGH I conducted a paired study, when I analyze my data, I can still PRETEND that my experiment result is a realization of UNPAIRED randomized experiment. Can I do this? Because if you truly did an unpaired random experiment, you can literal get the same outcome. Then I can just take the average of each group (ignore the pairing stuff) and take the difference of the two group mean. This is an unbiased estimator of E[Z]. For variance of my estimator, I just use... $\endgroup$
    – KevinKim
    Mar 8, 2018 at 3:49
  • $\begingroup$ @MichaelChernick the sample variance of group X and group Y and sum them up $\endgroup$
    – KevinKim
    Mar 8, 2018 at 3:49
7
$\begingroup$

Rather than pairing it is probably better to understand the underlying data model. If pairing is done to deal with uncontrolled heterogeneity, it is usually the case (except in twin studies) that the pairing only partially controls this source of variability and multiple regression would do better. This is because matching on continuous variables frequently results in residual variability because of not being able to do exact matching on such variables.

$\endgroup$
1
  • 2
    $\begingroup$ If we should all be doing regression, why do books on Experimental Design, like David Cox's book, stress the importance of pairing or grouping in biological experiments? Pairing avoids the hidden assumption of linear dependence entailed in regression. But perhaps there are other reasons: anyone?? $\endgroup$ Sep 27, 2012 at 15:35
6
$\begingroup$

The two tests (paired and unpaired) ask different questions so they can get different answers. Correct pairing nearly always is more powerful than unpaired - that's really the point of pairing. So, since you say the pairing is correct, it is likely that the p-value for your paired test is lower than for the same data unpaired. You could, of course, do both and see for yourself.

Therefore, the answer to your dilemma is substantive, not statistical. Is your pairing right?

Could you get a more significant result from random pairing than from a unpaired test? Let's see:

set.seed(2910110192)
x <- rnorm(100, 10, 2)
y <- rnorm(100, 10, 2)
t.test(x, y)
t.test(x, y, paired = T)

Yes you can, although here the difference is very small, the paired had a lower p. I ran that code several times. Not surprisingly, sometimes one p is lower, sometimes the other, but the difference was small in all cases. However, I am sure that in some situations the difference in p values could be large.

$\endgroup$
3
  • $\begingroup$ Thanks for the answer, but my question asked for systematic differences. Obviously, in a long run of x's and y's, x and y occasionally look as though they are very well-paired, and occasionally as though they have been deliberately badly paired. Surely it's a statistical question whether, on choosing x and y randomly, the distribution of p-values is the same on the two tests. I suppose it shouldn't be too difficult for someone who knows more theoretical statistics than I do to actually compute the two theoretical distributions of p-values. My guess is that they are the same. $\endgroup$ Sep 27, 2012 at 15:25
  • $\begingroup$ In the actual case I was involved in, the p-value for unpaired was around .04 and for paired .001. According to the critical biologist, we should be quoting .04. According to me, the improvement in p-value strongly indicates that our pairing was valid. I claim there is an objective question in statistics here, with an objective answer, and that it's not just a question of good biological judgement as to the validity of the particular pairing---the latter seems to be the opinion of Peter Flom and of the critical biologist. $\endgroup$ Sep 27, 2012 at 16:09
  • 1
    $\begingroup$ I think the statistics tells the story. Both results should be disclosed but as long as the data is correct and the correlation can be explained the paired test is more accurate because it takes the correlation into account. $\endgroup$ Oct 4, 2012 at 21:16
5
$\begingroup$

I now understand much better what was worrying me about paired versus unpaired t-tests, and associated p-values. Finding out has been an interesting journey, and there have been many surprises along the way. One surprise has resulted from an investigation of Michael's contribution. This is irreproachable in terms of practical advice. Moreover, he says what I think virtually all statisticians believe, and he has several upvotes to back this up. However, as a piece of theory, it is not literally correct. I discovered this by working out the formulas for the p-values, and then thinking carefully how to use the formulas to lead to counter-examples. I'm a mathematician by training, and the counter-example is a "mathematician's counter-example". It's not something you would come across in practical statistics, but it was the kind of thing I was trying to find out about when I asked my original question.

Here is the R-code that gives the counter-example:

vLength <- 10; meanDiff <-10^9; numSamples <- 3;
pv <- function(vLength,meanDiff) {
    X <- rnorm(vLength)
    Y <- X - meanDiff + rnorm(vLength,sd=0.0001)
    Paired <- t.test(X,Y,var.equal=T,paired=T)
    NotPaired <- t.test(X,Y,var.equal=T,paired=F)
    c(Paired$p.value,NotPaired$p.value,cov(X,Y))
}
ans <- replicate(numSamples,pv(vLength,meanDiff))

Note the following features: X and Y are two 10-tuples whose difference is huge and very nearly constant. To many significant figures, the correlation is 1.000.... The p-value for the unpaired test is around 10^40 times smaller than the p-value for the paired test. So this contradicts Michael's account, provided that one reads his account literally, mathematician-style. Here ends the part of my answer related to Michael's answer.


Here are the thoughts prompted by Peter's answer. During the discussion of my original question, I conjectured in a comment that two particular distributions of p-values that sound different are in fact the same. I can now prove this. What is more important is that the proof reveals the fundamental nature of a p-value, so fundamental that no text (that I've come across) bothers to explain. Maybe all professional statisticians know the secret, but to me, the definition of p-value always seemed strange and artificial. Before giving away the statistician's secret, let me specify the question.

Let $n>1$ and choose randomly and independently two random $n$-tuples from some normal distribution. There are two ways of getting a p-value from this choice. One is to use an unpaired t-test, and the other is to use a paired t-test. My conjecture was that the distribution of p-values that one gets is the same in the two cases. When I first started to think about it, I decided that this conjecture had been foolhardy and was false: the unpaired test is associated to a t-statistic on $2(n-1)$ degrees of freedom, and the paired test to a t-statistic on $n-1$ degrees of freedom. These two distributions are different, so how on earth could the associated distributions of p-values be the same? Only after much further thought did I realize that this obvious dismissal of my conjecture was too facile.

The answer comes from the following considerations. Suppose $f:(0,\infty)\to (0,\infty)$ is a continuous pdf (that is, its integral has value one). A change of coordinates converts the associated distribution into the uniform distribution on $[0,1]$. The formula is $$p=\int_t^\infty f(s)\,ds$$ and this much is explained in many texts. What the texts fail to point out in the context of p-values is that this is exactly the formula that gives the p-value from the t-statistic, when $f$ is the pdf for the t-distribution. (I'm trying to keep the discussion as simple as I can, because it really is simple. A fuller discussion would treat one-sided and two-sided t-tests slightly differently, factors of 2 might arise, and the t-statistic might lie in $(-\infty,\infty)$ instead of in $[0,\infty)$. I omit all that clutter.)

Exactly the same discussion applies when finding the p-value associated with any of the other standard distributions in statistics. Once again, if the data is randomly distributed (this time according to some different distribution), then the resulting p-values will be distributed uniformly in $[0,1]$.

How does this apply to our paired and unpaired t-tests? The point is in the paired t-test, with samples chosen independently and randomly, as in my code above, the value of t does indeed follow a t-distribution (with $n-1$ degrees of freedom). So the p-values that result from replicating the choice of X and Y many times follow the uniform distribution on $[0,1]$. The same is true for the unpaired t-test, though this time the t-distribution has $2(n-1)$ degrees of freedom. Nevertheless, the p-values that result also have a uniform distribution on $[0,1]$, by the general argument I gave above. If Peter's code above is applied to determine p-values, then we get two distinct methods of drawing a random sample from the uniform distribution on $[0,1]$. However the two answers are not independent.

$\endgroup$
11
  • $\begingroup$ I don't think the p-value has any mysterious secets to it. Some people have a difficult time with it. It is the probability of observing a value as extereme or more extreme than what was actually observed when the null hypothesis is TRUE. I think you had that right in one of your formulas. I do think you stated that p-values are uniformly distributed. Yes I agree with that when the null hypothesis is true. Keep in mind that with your t test the null hypothesis may not be true. Then the p-value is not uniform. It should be concentrated closer to 0. $\endgroup$ Oct 4, 2012 at 21:25
  • $\begingroup$ Secondly we are talking about two different test statistics. One is based on pairing and one in not in your example. Whether I mentioned it in my answer or not the unpaired t test has a central t distribution with 2n-2 degrees of freedom while the corresponding t distribution for the paired t test has n-1 degrees of freedom. So the one with the larger number of degrees of freedom is closer to the standard normal distribution than the other. Does that matter when you applies these tests to real data? No! Not when n is reasonably large. $\endgroup$ Oct 4, 2012 at 21:32
  • $\begingroup$ As a side note a limitation of the paired test is requiring equal sample size which you should have if all the data can be paired. But the unpaired test is valid with unequal sample sizes. So in general the unpaired test has n+m-2 degrees of freedom. $\endgroup$ Oct 4, 2012 at 21:34
  • $\begingroup$ Your answer is long and abstract and I tried to wade through it but I didn't understand the counterexample. I just don't see where you take the null hypothesis and the real data into account. The observed p-value is the integral of the appropriate t distribution for the test statistic given the data. You compare those numbers for the two t distributions and the same common data set. If you condition on the observed data these uniform distributions play no role. I am sorry but I don't see that your answer really answers your question. $\endgroup$ Oct 4, 2012 at 21:42
  • $\begingroup$ Michael: just concentrate on the R-code I gave. It only takes a second to run. The null hypothesis is that X and Y come from the same normal distribution, which is, of course, wildly false in my case. In my example Cov(X,Y)>0 and nevertheless the unpaired test gives more significance than the paired test. $\endgroup$ Oct 5, 2012 at 8:28
1
$\begingroup$

I would offer another perspective. Often, pairing is done do reduce bias. Suppose that you are interested in whether exposure E is a risk factor for a continuous outcome Y. For each E+ subject, you get an age and sex matched subject who is E-. Now, we could do either a paired t-test or an unpaired t-test. I think we should account for matching explicitly and conduct a paired t-test. It is more principled in that it takes the design into account. Whether to take matching into account in the analysis is an issue of the bias-variance tradeoff. Accounting for matching in the analysis provides more protection against bias, but can increase variance. Doing an unpaired t-test may be more efficient, but it would not provide any protection against bias.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.