Once again: paired versus unpaired t-tests This is much the same question that I asked a few weeks ago, but I hope to explain myself more clearly this time.
I start with 40 mice. I use my own scheme, based on my own ideas for how to make the two mice in a pair as similar as possible in terms of the experiment I am about to do. Pairing the 40 mice gives me 20 pairs; from now on the pair structure is unchanged. Using a random number generator, I choose one mouse from each pair, selected to be given the drug A; the other mouse in each pair is given drug B. From now on, the two mice in a pair are treated as similarly as possible in all respects, except that they are given the two different drugs A and B. For example they are always fed at the same time. We also take the precaution of doing the experiment in a blinded fashion, that is, arranging the experiment so that only one person X knows which mice are getting A and which B, and so that X has no other role in the experiment. The experiment has a numerical outcome, namely the cholesterol concentration in the blood at the end of the experiment, so we get 20 numbers. Let's assume that two distributions involved are approximately normal, with the same variance. The null hypothesis is that the means are the same. The mean for mice given drug A mice turns out to be lower than the mean for mice given drug B. Moreover, a two-sided paired t-test gives me a p-value of 0.005, which is eminently publishable, and an two-sided unpaired t-test gives a p-value of 0.06, which would make the work unpublishable according to criteria that are standard in biology publications.
An eminent expert in the application field says that my pairing scheme is "not biological", and that the "correct" p-value should therefore be 0.06.
My own reaction is:


*

*that it is possible that the result is a fluke, so maybe the experiment should be repeated, particularly if the conclusions seem unlikely to experts.

*Barring statistical flukes, what has been shown is that drug A is more effective at lowering the blood cholesterol in a mouse than drug B, and this result is statistically significant. 

*Barring flukes, the experiment shows that my very own pairing method has a sound biological basis, though an understanding of why the method is sound may still be unavailable.

*Unless something is wrong with the experiment, other than the pairing scheme, the eminent expert is wrong. 


What is the response of the Cross-Validated community? I'm not asking for practical advice about what to do with the experimental results---the paper is already published. I'm wanting to make quite sure that what I say is, in principle, correct, as it will affect my advice for future experiments.
 A: The pairing has two different aspects that need to be considered. First, how were the pairs selected? Several people have asked about this. Additionally, pairing controls for differences in the experimental manipulations. Maybe different people handled the animals on different days, so some pairs were handled more gently than others. Or some pairs were exposed to colder temperatures. Or some pairs were given a different lot of food than others. Doing a paired t test because the pairing controlled for subtle differences in experimental handling is valid, even if the pairs were originally chosen randomly. 
A: There are some angles on this to consider. First off, your study result is dependent on your pairing scheme, which you didn't describe in detail. If it's sensible this won't matter much, but there are ways to pair samples/patients to provoke differences. An obvious example would be to pair males with academic background and working class females and then report on a difference in IQ due to gender. So your result is conditional on the pairing scheme.
This being said, if you keep this in mind when interpreting your result any pairing is legal. This includes totally random pairing - in this case you basically ask the same question as the unpaired t-test. Remember that if you look at the differences of random variables X and Y you get:
$$
Var(X-Y)=Var(X)+Var(Y) - 2*Cov(X,Y).
$$
This basically tells you that - compared to the unpaired t-test - the paired t-test has higher power if you have positive correlation between the values of each pair and lower paired if you have negative correlation. So you can't just inflate your signifances by deciding to run only paired t-tests, as long as you decided on the pairing and the test in advance. You can obviously inflate them by running both and reporting the better one. 
Another tricky thing to consider might be the distribution of the values. The distribution within the groups and the distribution of the paired differences might be totally different. For example, ff there's a hidden factor that decides whether the treatments works or not, the distribution of differences might actually be bimodal. 
I do disagree with the point by @mbq above that the unpaired t-test is safer. With the paired test you have to make sure that you have as homogenous pairs as possible, with the unpaired you still have to make sure that the treatment group is not already healthier/fitter than the control group or vice versa. In both cases, you have to make that certain - and which samples sizes this small simple assigning treatments by chance can already easily result in a skewed distribution.
A: This is an interesting question. As I understand it, what you really want to know is whether or not your pairing method is reasonable to use. Points 1,2 and 4 from your interpretation follow naturally if point 3 is valid, so I'm not going to address those.
I'll start by restating your experimental procedure, as I understand it, so that we can make sure we're talking about the same thing.
Your procedure is as follows:


*

*Define a measure P on pairs of mice, such that P(m1,m2) is small if m1 and m2 are similar, and large otherwise (for some values of small and large).

*Let M be a set of 20 randomly selected mice.

*Define M' to be the set of 10 pairs (m1,m2) $\in M$ such that $\sum_{m \in M'} P(m)$ is minimized.

*At random, give drug A to one member of each pair of $M'$, and drug B to the other member, while treating both members identically otherwise.

*Measure the cholesterol level $C(m)$ of each mouse. 

*Perform a paired t-test with null hypothesis $\hat C(m_1) = \hat C( m_2)$.


And, in essence, the expert critic objects to your choice of $P$, and thus to your having used a paired test. Is this correct?
Assuming I've understood you correctly, then I think his critique is reasonable. Since you have used a paired test, as Erik pointed out above, a pairing method that is unreasonable will lead to a p-value that is more or less meaningless. For example, if your pairing metric were especially silly, and assigned similarly as the inverse of age difference, then the mice within a pair would have extremely different initial cholesterol levels. With a small sample, it is then easy to believe that a small p-value could result from a chance difference in the group assigned to each member of a pair. In this case, it is more reasonable to use the unpaired test.
That said, if your method is reasonable (e.g. you tested initial cholesterol levels) then I think this is probably fine. So it seems like the real issue is whether you can justify your choice of pairings. 
