4
$\begingroup$

If I want to compare two sets of measurements, i.e., how much their means differ or how much the sets differ, I would say I have two options:

  1. Paired difference t-test: Calculate the differences to get the change score for each element and then use the set of differences to build the t statistic (with the mean and standard deviation of the differences) and compare with the t-distribution. The degrees of freedom would be n-1.

  2. Similar approach, but calculating means and standard deviation separately for both samples. Build the the statistic using the two means and standard deviations. The degrees of freedom would be 2n-2.

What are the differences between these two approaches?

$\endgroup$
4
$\begingroup$

To readers : please note the hierarchy of the answer :-)

Suppose $X\sim N(\mu_x,\sigma_x^2)$ and $Y\sim N(\mu_y,\sigma_y^2)$. For simplicity, suppose $\sigma_x^2=\sigma_y^2=\sigma^2$, which is unknown. Suppose the two samples are $\mathbb{X}=\{X_1,\dots,X_m\}$ and $\mathbb{Y}=\{Y_1,\dots,Y_n\}$. We are testing $H_0: \mu_x-\mu_y=0$

  1. when $m\neq n$
    • paired t-test is not applicable.
    • 2-sample t-test is applicable if $\mathbb{X}$ and $\mathbb{Y}$ are independent.
  2. when $m=n$

    1. if $\mathbb{X}$ and $\mathbb{Y}$ are matched and thus not independent, then
      • paired t-test is applicable
      • 2-sample t-test is not applicable as it assumes independence
    2. if $\mathbb{X}$ and $\mathbb{Y}$ are independent, then

      • both paired t-test and 2-sample t-test are applicable.
      • paired t-test
        • Let $Z_i=X_i-Y_i$, where $i=1,2,\dots,n$, and $X_i$ and $Y_i$ are $i$-th observations in $\mathbb{X}$ and $\mathbb{Y}$ respectively. Then test statistic is $t_1=\frac{\bar{Z}}{S_z/\sqrt{n}} = \frac{\bar{X}-\bar{Y}}{S_z/\sqrt{n}}$, where $S_z$ is the sample standard deviation of $Z_i$'s. When $H_0$ is true, $t_1 \sim t(n-1)$. Note that the value of $Z_i$'s depends on the ordering of observations in $\mathbb{X}$ and $\mathbb{Y}$, so does $S_z$. Since $X_i$ and $Y_i$ are not paired, we can arbitrarily re-order the observations in each of $\mathbb{X}$ and $\mathbb{Y}$, and get different values of $Z_i$'s, $S_z$ and thus $t_1$. This is the most obvious disadvantage of applying paired test to unpaired data. To make the test "objective", let's order the observations in $\mathbb{X}$ and $\mathbb{Y}$ in a completely random manner.
      • 2-sample t-test
        • test statistic is $t_2= \frac{\bar{X}-\bar{Y}}{\sqrt{S_x^2+S_y^2}/\sqrt{n}}$. When $H_0$ is true, $t_2 \sim t(2n-2)$.
      • What is the difference between $t_1$ and $t_2$?
        • Theoretically, both tests work, but they have different derivations. $t_1$ is based on one normal random variable $\bar{Z}$ and one $\chi^2_{n-1}$ random variable $(n-1)S_z^2$ which is independent of $\bar{Z}$. $t_2$ is based on difference between two independent normal variables $\bar{X}$ and $\bar{Y}$, and addition of two independent $\chi^2_{n-1}$ random variables $(n-1)S_x^2$ and $(n-1)S_y^2$ which are both independent of $\bar{X}$ and $\bar{Y}$.
        • What is the relationship between $S_x^2$, $S_y^2$ and $S_z^2$? $S_z^2 = \frac{1}{n-1}\sum\limits_{i=1}^{n}(Z_i-\bar{Z})^2$ $ = \frac{1}{n-1}\sum\limits_{i=1}^{n}(X_i-Y_i-\bar{X}+\bar{Y})^2$ $ = \frac{1}{n-1}\sum\limits_{i=1}^{n}[(X_i-\bar{X})-(Y_i-\bar{Y})]^2$ $ = \frac{1}{n-1}\sum\limits_{i=1}^{n}(X_i-\bar{X})^2 + \frac{1}{n-1}\sum\limits_{i=1}^{n}(Y_i-\bar{Y})^2 - 2\frac{1}{n-1}\sum\limits_{i=1}^{n}(X_i-\bar{X})(Y_i-\bar{Y})$ $ = S_x^2 + S_y^2 - 2S_{xy}$, where $S_{xy}$ is the sample covariance between $\mathbb{X}$ and $\mathbb{Y}$, which again depends on the ordering of observations in $\mathbb{X}$ and $\mathbb{Y}$.
      • Which test should we choose?

        • key word : statistical power of test, which is the probability for a test to reject $H_0$ when $H_0$ is false.
        • Situation 1: If $S_{xy} = 0$, then $S_z^2 = S_x^2 + S_y^2$ and $|t_1| = |t_2|$. Since $t_2$ has larger degree of freedom, the corresponding p-value is smaller than $t_1$. Hence, when $H_0$ is false, 2-sample t-test has larger statistical power (making you more confident in rejecting $H_0$) than paired t-test. Since $X$ and $Y$ are independent, we know their population covariance is $\sigma_{xy} = 0$. When the observations in $\mathbb{X}$ and $\mathbb{Y}$ are randomly ordered, intuitively, we have a good reason to believe that the sample covariance $S_{xy}$ is not far from the population covariance $\sigma_{xy}=0$, and thus believe $S_{xy} \approx 0$ is the most likely situation.
        • Situation 2: If $S_{xy} < 0$ (we can always achieve this by cheating: order the observations in $\mathbb{X}$ in ascending order and $\mathbb{Y}$ in descending order, which is the reason why we should use random ordering), then $S_z^2 > S_x^2 + S_y^2$ and $|t_1| < |t_2|$. Since $t_2$ also has larger degree of freedom than $t_1$, the 2-sample t-test has smaller p-value than paired t-test. When $H_0$ is false, 2-sample t-test has larger statistical power.
        • Situation 3: If $S_{xy} > 0$ (we can always achieve this by cheating: order the observations in both $\mathbb{X}$ and $\mathbb{Y}$ in ascending order, which is the reason why we should use random ordering), then $S_z^2 < S_x^2 + S_y^2$ and $|t_1| > |t_2|$. Now $|t_1| > |t_2|$ but $t_2$ has more degrees of freedom than $t_1$, so it is hard to say which test statistic gives larger p-value or which test should be preferred.
        • Conclusion: Considering all 3 situations, overall, when $X$ and $Y$ are independent, it is better to choose $t_2$ (two-sample t-test) rather than $t_1$ (paired t-test) if we use statistical power as the criterion. Especially, since $S_{xy} \approx 0$ is the most likely situation, we simply choose the test having larger power in situation 1, which is two-sample t-test.
        • A simulation study is conducted to compare the two test procedures in term of statistical power. Let $X\sim N(0, 1)$, $y\sim N(\mu_y, 1)$, let $n=10$ and let $\alpha = 0.05$. The x-axis represents the true value of $\mu_y$ ($H_0: \mu_x = \mu_y$ is true when $\mu_y = 0$), and the y-axis shows the percentages of rejecting $H_0$ using the 2 test procedures. As you can see, when $H_0$ is true ($\mu_y=0$), the two procedures have almost the same type I error; when $H_0$ is false, 2-sample t-test has more chance of rejecting $H_0$.comparison between paired t-test and 2-sample t-test

          n.rep <- 10000
          n <- 10
          alpha <- 0.05
          mu.y.seq <- seq(0, 2, 0.1)
          len.mu.y.seq <- length(mu.y.seq)
          pct.rej.mat <- matrix(0, nrow = len.mu.y.seq, ncol = 2)
          for(mu.y.index in 1:len.mu.y.seq){
            n.rej.1 <- 0
            n.rej.2 <- 0
            for(rep in 1:n.rep){
              set.seed(rep)
              X <- rnorm(n = n, mean = 0)
              Y <- rnorm(n = n, mean = mu.y.seq[mu.y.index])
              Z <- X - Y
              t1 <- t.test(x = Z)
              t2 <- t.test(x = X, y = Y)
              if(t1$p.value < alpha) n.rej.1 <- n.rej.1 + 1
          if(t2$p.value < alpha) n.rej.2 <- n.rej.2 + 1
          
            }
            pct.rej.mat[mu.y.index, ] <- c(n.rej.1, n.rej.2)/n.rep
          }
          plot(pct.rej.mat[ ,1] ~ mu.y.seq, ylim = c(0, 1), xlab = expression(paste(mu[y])), ylab = "percentage of rejecting H0", main = "comparing paired t-test with 2-sample t-test for independent X and Y")
          lines(pct.rej.mat[ ,1] ~ mu.y.seq)
          points(pct.rej.mat[ ,2] ~ mu.y.seq, col = "red")
          lines(pct.rej.mat[ ,2] ~ mu.y.seq, col = "red")
          legend(x = c(0,0.5), y = c(0.84, 1), legend = c("paired t-test","2-sample t-test"), col = c("black", "red"), pch = c(1,1) )
          
$\endgroup$
  • $\begingroup$ I'm not sure how the calculations factor into the decision about which test to choose: the decision should be based around whether the data are in fact paired observations or not? $\endgroup$ – James Stanley Mar 2 '15 at 20:24
  • 1
    $\begingroup$ Also, $t_1\approx t_2$ would only hold if X and Y are in fact independent. While you give this as one of your assumptions, if you have paired measurements there should be at least some dependence due to correlation between X and Y measured in the same element [sampling unit] (which is reflected in the variance formula for the paired estimate), and the paired t-test will return a higher t-value than running an unpaired t-test on the same data... $\endgroup$ – James Stanley Mar 2 '15 at 20:29
  • $\begingroup$ @JamesStanley Please notice the hierarchical structure of the answer. $\endgroup$ – JellicleCat Mar 2 '15 at 20:32
  • 1
    $\begingroup$ @JellicleCat I still think that the decision is more simply based around your initial two subheadings under point #2 (that is, whether data are paired or unpaired.) Note also that imposing arbitrary 'pairing' on unpaired data means that the estimate of $S_z^2$ is dependent on the ordering of 'pairings' selected, which is a second reason not to select unpaired tests. ($S_{xy} \approx 0$ is the expected long-run value of $S_{xy}$, rather than the value in any given arbitrary pairing.) see stats.stackexchange.com/q/47453/16974 for an R example of this. $\endgroup$ – James Stanley Mar 2 '15 at 20:55
  • 1
    $\begingroup$ The case not considered in this answer is what happens if you do an independent test on paired data. This is suboptimal, but sometimes you do not have the pairing information, e.g. a before and after measurement collected from an anonymous survey, you know the design is paired, but don't know which values pair together. In this case if the correlation is positive then the independent test is conservative (p-values will be larger, confidence intervals will be wider) and so you can still make reasonable conclusions in this case. $\endgroup$ – Greg Snow Mar 2 '15 at 23:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.