Consider the the situation you want to know whether a new medicine will help to reduce the blood pressure.

Assume now you do a random draw from your target population, say you have 20 people in the lab, then you do randomized assignment, let's say it leads to 10 people in treatment group and 10 people in control group. Let's say there is no placebo effect. So you given the 10 people in the treatment group the new medicine, then after some time, you measure the blood pressure of two group of people, then you conduct independent unpaired t-test.

Alternatively, you could do the following: you first measure the blood pressure of all these 20 people, then you give all of them the new medicine, then after some time you take their blood pressure again, then you run a paired t-test.

I am questioning the 2nd method in the following way:

Under Rubin's potential outcome framework, the population will have 2 potential outcome distributions, one is the distribution of current blood pressure denoted by $F_c$, the other is the distribution that forces all people in the population take the new medicine and then take their blood pressure,denoted by $F_t$. The logic of Method 1 is theoretically correct, i.e., the 10 people from control group is a representative sample of the distribution of $F_c$ and the 10 people from the treatment group, is a representative sample of the distribution of $F_t$, then using the sample mean, we can make inference about the true mean difference of $F_c$ and $F_t$. Because the treatment group sample and control group sample are independent, in particular, they are all different people.

Now let's look at Method 2, I agree that these 20 people is a representative sample of $F_c$, now here comes the problem, after you give all these 20 people the new medicine, these 20 data points get mapped into the data points that form $F_t$. However, there is no guarantee that "these 20 data" is a representative of the $F_t$! Hence, we cannot draw any conclusion about the true mean difference between $F_c$ and $F_t$ using the sample in Method 2. Did I miss something?

I understand that for Method 2, you perform paired-sample t test, which will in general has less variability as the unpaired t-test, which sounds like pretty attractive. In practice, I don't know whether Method 1 or Method 2, which is more popular. What is the disadvantage of Method 2 comparing to Method 1? Why not just abandon Method 1 and all use Method 2?

  • $\begingroup$ For the FDA controlled clinical trials are the "gold" standard. So they would want the company trying to get a drug approved to use method 1. When there is an existing drug on the market you need to show the drug is at least as effective and safe as the drug already on market. To get a fair comparison you would want to show for a current population the new drug is as good as the control drug. This is achieved by randomizing patients to these two treatments. $\endgroup$ – Michael Chernick Dec 9 '16 at 22:08
  • 1
    $\begingroup$ "However, there is no guarantee that 'these 20 data' is a representative of the $F_t$!" — Huh? Why not? You gave these subjects the medicine and measured their blood pressure. $\endgroup$ – Kodiologist Dec 9 '16 at 22:47
  • $\begingroup$ @Kodiologist I kind of figure out myself, correct me if I am wrong. I should look at a new conceptual distribution which is generated by the point-to-point difference of F_t and F_c. Because the points in the support of F_t and F_c are linked by each people. Then these 20 people should be viewed as a representative sample of this difference distribution. Then the goal becomes to use sample (20 people) to infer whether the mean of this difference distribution is 0 or not $\endgroup$ – KevinKim Dec 10 '16 at 0:04
  • $\begingroup$ @MichaelChernick I see. If I want to compare the effect of drug A and drug B, then Method 2 does not work, since I could not first let all my sample take drug A and measure their metrics, then let them all take drug B and measure their metrics again. There may be interaction with first take drug A and then take drug B $\endgroup$ – KevinKim Dec 11 '16 at 14:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.