4
$\begingroup$

This is much the same question that I asked a few weeks ago, but I hope to explain myself more clearly this time.

I start with 40 mice. I use my own scheme, based on my own ideas for how to make the two mice in a pair as similar as possible in terms of the experiment I am about to do. Pairing the 40 mice gives me 20 pairs; from now on the pair structure is unchanged. Using a random number generator, I choose one mouse from each pair, selected to be given the drug A; the other mouse in each pair is given drug B. From now on, the two mice in a pair are treated as similarly as possible in all respects, except that they are given the two different drugs A and B. For example they are always fed at the same time. We also take the precaution of doing the experiment in a blinded fashion, that is, arranging the experiment so that only one person X knows which mice are getting A and which B, and so that X has no other role in the experiment. The experiment has a numerical outcome, namely the cholesterol concentration in the blood at the end of the experiment, so we get 20 numbers. Let's assume that two distributions involved are approximately normal, with the same variance. The null hypothesis is that the means are the same. The mean for mice given drug A mice turns out to be lower than the mean for mice given drug B. Moreover, a two-sided paired t-test gives me a p-value of 0.005, which is eminently publishable, and an two-sided unpaired t-test gives a p-value of 0.06, which would make the work unpublishable according to criteria that are standard in biology publications.

An eminent expert in the application field says that my pairing scheme is "not biological", and that the "correct" p-value should therefore be 0.06.

My own reaction is:

  1. that it is possible that the result is a fluke, so maybe the experiment should be repeated, particularly if the conclusions seem unlikely to experts.
  2. Barring statistical flukes, what has been shown is that drug A is more effective at lowering the blood cholesterol in a mouse than drug B, and this result is statistically significant.
  3. Barring flukes, the experiment shows that my very own pairing method has a sound biological basis, though an understanding of why the method is sound may still be unavailable.
  4. Unless something is wrong with the experiment, other than the pairing scheme, the eminent expert is wrong.

What is the response of the Cross-Validated community? I'm not asking for practical advice about what to do with the experimental results---the paper is already published. I'm wanting to make quite sure that what I say is, in principle, correct, as it will affect my advice for future experiments.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ The question here is whether all mouse are homogeneous before your experiment. If there is a significant variation in the untouched cholesterol metabolism or in your drug metabolism, it can spoil the paired test assumptions -- you've could be just lucky and assign mostly "fitter" or "more responsive" mice to be treated. You can probably investigate it by looking at the variation over the control mice, but the unpaired test is way safer. $\endgroup$
    – user88
    Oct 15, 2012 at 10:36

3 Answers 3

Reset to default
1
$\begingroup$

The pairing has two different aspects that need to be considered. First, how were the pairs selected? Several people have asked about this. Additionally, pairing controls for differences in the experimental manipulations. Maybe different people handled the animals on different days, so some pairs were handled more gently than others. Or some pairs were exposed to colder temperatures. Or some pairs were given a different lot of food than others. Doing a paired t test because the pairing controlled for subtle differences in experimental handling is valid, even if the pairs were originally chosen randomly.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ These are good points, and, in my opinion, are very relevant in biology. $\endgroup$
    – David Epstein
    Oct 17, 2012 at 15:32
0
$\begingroup$

This is an interesting question. As I understand it, what you really want to know is whether or not your pairing method is reasonable to use. Points 1,2 and 4 from your interpretation follow naturally if point 3 is valid, so I'm not going to address those.

I'll start by restating your experimental procedure, as I understand it, so that we can make sure we're talking about the same thing.

Your procedure is as follows:

  1. Define a measure P on pairs of mice, such that P(m1,m2) is small if m1 and m2 are similar, and large otherwise (for some values of small and large).
  2. Let M be a set of 20 randomly selected mice.
  3. Define M' to be the set of 10 pairs (m1,m2) $\in M$ such that $\sum_{m \in M'} P(m)$ is minimized.
  4. At random, give drug A to one member of each pair of $M'$, and drug B to the other member, while treating both members identically otherwise.
  5. Measure the cholesterol level $C(m)$ of each mouse.
  6. Perform a paired t-test with null hypothesis $\hat C(m_1) = \hat C( m_2)$.

And, in essence, the expert critic objects to your choice of $P$, and thus to your having used a paired test. Is this correct?

Assuming I've understood you correctly, then I think his critique is reasonable. Since you have used a paired test, as Erik pointed out above, a pairing method that is unreasonable will lead to a p-value that is more or less meaningless. For example, if your pairing metric were especially silly, and assigned similarly as the inverse of age difference, then the mice within a pair would have extremely different initial cholesterol levels. With a small sample, it is then easy to believe that a small p-value could result from a chance difference in the group assigned to each member of a pair. In this case, it is more reasonable to use the unpaired test.

That said, if your method is reasonable (e.g. you tested initial cholesterol levels) then I think this is probably fine. So it seems like the real issue is whether you can justify your choice of pairings.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Your description of the method is good. Wouldn't your objection of a "chance difference" apply equally to an unpaired t-test, with equally meaningless p-values? I'm asking whether the paired t-test can give a worse answer if the measure P is incompetently chosen. If everything is random normal, then the p-values will be uniformly distributed in [0,1] in both paired and unpaired tests. I've already edited once, trying to remove the "small sample" problem. I'll now edit it again in the same direction. Are small samples more of a problem for paired tests than for unpaired tests? $\endgroup$
    – David Epstein
    Oct 15, 2012 at 15:16
  • $\begingroup$ Hmm. In light of your comment I've thought this through again. The short answer is that yes, if you choose a really crazy P, then the unpaired test can give a lower p-value than the paired one. As Erik said, if the correlation between elements in a pair is negative, then the unpaired test becomes an overly liberal estimator. That said, I think that your randomizing of the order will probably prevent this. So the main problem is actually the small sample, but even this is not so bad. $\endgroup$
    – John Doucette
    Oct 15, 2012 at 16:43
  • $\begingroup$ hmm. Thinking more. Might edit my response in a moment. $\endgroup$
    – John Doucette
    Oct 15, 2012 at 16:45
  • $\begingroup$ I'm actually not at all sure about this. I'll leave my answer up as a talking point perhaps. $\endgroup$
    – John Doucette
    Oct 15, 2012 at 16:55
  • 1
    $\begingroup$ I believe that a really crazy P will give a lower p-value half the time and a higher p-value half the time, and, moreover, the two p-values will be pretty near each other for data obtained in practice, very near if there really is a substantial difference in the means. The bottom line, I believe, is a general rule of thumb, that, provided the pairing is done in the way I specified in my question, one can, in practice, only profit from a paired t-test, and this is an entirely legitimate profit. Can anyone give me a counter-example? $\endgroup$
    – David Epstein
    Oct 15, 2012 at 17:32
0
$\begingroup$

There are some angles on this to consider. First off, your study result is dependent on your pairing scheme, which you didn't describe in detail. If it's sensible this won't matter much, but there are ways to pair samples/patients to provoke differences. An obvious example would be to pair males with academic background and working class females and then report on a difference in IQ due to gender. So your result is conditional on the pairing scheme.

This being said, if you keep this in mind when interpreting your result any pairing is legal. This includes totally random pairing - in this case you basically ask the same question as the unpaired t-test. Remember that if you look at the differences of random variables X and Y you get:

$$ Var(X-Y)=Var(X)+Var(Y) - 2*Cov(X,Y). $$

This basically tells you that - compared to the unpaired t-test - the paired t-test has higher power if you have positive correlation between the values of each pair and lower paired if you have negative correlation. So you can't just inflate your signifances by deciding to run only paired t-tests, as long as you decided on the pairing and the test in advance. You can obviously inflate them by running both and reporting the better one.

Another tricky thing to consider might be the distribution of the values. The distribution within the groups and the distribution of the paired differences might be totally different. For example, ff there's a hidden factor that decides whether the treatments works or not, the distribution of differences might actually be bimodal.

I do disagree with the point by @mbq above that the unpaired t-test is safer. With the paired test you have to make sure that you have as homogenous pairs as possible, with the unpaired you still have to make sure that the treatment group is not already healthier/fitter than the control group or vice versa. In both cases, you have to make that certain - and which samples sizes this small simple assigning treatments by chance can already easily result in a skewed distribution.

Improve this answer
$\endgroup$
6
  • $\begingroup$ I carefully described my procedure in the question. If I pair each female with a university background with a male who left school at 15, then I have to SUBSEQUENTLY choose one of them AT RANDOM. So I don't see how I could make the fallacious report you suggest in your first paragraph, particularly as I would have to describe my procedure. There's only one chance in 32 that each group is same sex. To make sure that examples such as the one you just gave are not allowed, I will edit my question to increase the number of subjects. $\endgroup$
    – David Epstein
    Oct 15, 2012 at 10:53
  • $\begingroup$ There is a reason that I didn't specify my pairing method in my question. Everyone else may think that the pairing method is not sensible, but I'm PROVING that it is sensible by using a paired t-test. I'm asking about the principle of how these tests can be used, not about the details of one particular experiment. It would always be open to a sceptic about my results to say I have paired badly. I'm saying that such criticism is INEVITABLY unfair. $\endgroup$
    – David Epstein
    Oct 15, 2012 at 11:05
  • $\begingroup$ As far as practical advice is concerned, I agree with the point about positive and negative correlations. However, from the point of view of theory, it's incorrect. In my answer at stats.stackexchange.com/questions/38102/… I gave 5 lines of R-code that provides a conclusive counter-example. Admittedly the example is totally wild from the numerical point of view, but it's fine in theory. In my example, the correlation of X and Y is 1.000000 to a large number of decimal places. Despite this, the unpaired t-test is more powerful than the paired test. $\endgroup$
    – David Epstein
    Oct 15, 2012 at 11:36
  • $\begingroup$ Yes, I am actually aware of that. If you go far enough into the tails of the t-distribution the fewer degrees of freedom of the paired test outweigh the higher actual value of the t-statistic. However, it is usually not a large factor in practice, so I decided to ignore it for the purpose of the post. BTW I would beware trusting the numerical accuracy of results of this magnitude as well. $\endgroup$
    – Erik
    Oct 15, 2012 at 13:36
  • 1
    $\begingroup$ More relevantly, I do not see how you PROVE that the pairing is sensible by using the test. Showing that with that pairing you get a significant result is certainly not a proof. If I had to reason for the sense of a pairing, I would show that the inner group variability is significantly reduced compared to the overall variability. $\endgroup$
    – Erik
    Oct 15, 2012 at 13:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.