# Comparing two, or more, independent paired t-tests

• Can we compare the results from two, or more, independent paired t-tests?

For example: I want to test if drug 1 and drug 2 are effective to reduce weight. I have a control group (that will consume a placebo drug), one test group for drug 1, and another test group for drug2, all of the same size.

Instead of doing ANOVA, I want to make three independent paired t-tests, i.e, one paired t-test for each one of the groups (measuring weight before and weight after). Say that for the paired t-test for the control group I see a mean weight loss of 0.6kg, for the test group (drug 1) there is a mean weight loss of 1.3kg and for the other test group (drug 2) there is a mean weight loss of 1.4kg.

Can I compare the outputs from the three paired t-tests? From these independent tests can I already say that drug 1 and drug 2 are more effective than the placebo drug?

Edit: The problem with repeated measures ANOVA, is that if I added a variable time with $$t_1$$ (before) and $$t_2$$ (after), the model would assume that $$t_1$$ was already depending on the group. For example, if my data looks like this:

id   drug    t1-weight t2-weight
1     1        3.4       3.1
2   placebo    3.8       3.7
3     2        4.0       3.7
4   placebo    3.3       3.2
5     1        4.4       4.4
...   ...       ...       ...



Doesn't the ANOVA think that the value measured at $$t_1$$ is already being affected by the drug type? This is a problem because at $$t_1$$ none of the drugs are affecting because that measure is BEFORE the drug administration. I think the only way here is to do paired t-tests. However, how can I compare them?

This problem would also be more complex if I needed to add another variable, for example a variable for gender (male and female) to check if gender also affects the treatment?

Using the difference between t2 and t1 as feature should work; one can use ANOVA to test the equality of the means.

Note that there are post-hoc tests that allow you to get more insight into the differences, if any, found. This allows you to identify which means deviate. These post-hoc tests take into account that multiple test are being made; i.e. 'Bonferroni test' included.

You can also use a two way ANOVA if you want to add gender as second variable.

Would you want to add more variables, you could try to setup the tests as a hierarchical linear regression problem with dummy variables. Starting out with a model with a single mean, and comparing this model to one that has two dummy variables for the experimental conditions would give you the answer you want. The smaller and the larger model can be compared because the models are considered hierarchical; i.e. the larger model extends the smaller model.

Using an extra dummy for gender, and potentially dummies for interaction effects for gender x treatment, one can squeeze a lot of information out of the data. Note that one has to apply some Bonferroni type correction in this case. One would like to do some kind of power analysis upfront to see if the sample size sustains multiple testing without loosing to much power.

Happy testing!

Dunnett's test is the canonical test for a one-way ANOVA of various treatments versus control. It has slightly more statistical power compared to multiple independent t-tests with Bonferoni corrections under the baseline assumptions.

See the Wikipedia entry on Dunnett's test: https://en.wikipedia.org/wiki/Dunnett%27s_test

Any good design of experiments text book will have a discussion of a one-way ANOVA and Dunnett's test. See this STATS 503 class: https://online.stat.psu.edu/stat503/lesson/3/3.1

Here is an example analysis in R code:

require(DescTools)
require(ggplot2)

# Simulate Data
set.seed(184873)
NperTreat <- 20
dat <- data.frame(group = factor(rep(c("Control", "Treat1", "Treat2"), each = NperTreat)),
t1 = rnorm(3*NperTreat, 74.7, 15.46),
t2 = c(rnorm(NperTreat, 74.7, 15.46),
rnorm(NperTreat, 74.7 - 2, 15.46),
rnorm(NperTreat, 74.7 - 15, 15.46)))
dat$$weight_difference <- dat$$t2 - dat$t1 ggplot(dat, aes(x = group, y = weight_difference)) + geom_boxplot()  mod1 <- aov(weight_difference ~ group, data = dat) summary(mod1) #> Df Sum Sq Mean Sq F value Pr(>F) #> group 2 6043 3022 4.827 0.0116 * #> Residuals 57 35684 626 #> --- #> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 # There is at least one difference between the group means p = 0.00116 DescTools::DunnettTest(dat$$weight_difference, g = dat$$group, control = "Control") #> #> Dunnett's test for comparing several treatments with a control : #> 95% family-wise confidence level #> #>$Control
#>                       diff    lwr.ci    upr.ci   pval
#> Treat1-Control  -0.9113978 -18.85869 17.035898 0.9903
#> Treat2-Control -21.7305135 -39.67781 -3.783218 0.0153 *
#>
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

# there is insufficient evidence to conclude treatment1 is differnt from control p = 0.9903
# treatment 2 is different from control by an estimated 21 kg p = 0.0153

################################################################################

# Two independent t-tests have less power than Dunnett's to detect a difference
#   a test at alpha =  0.05 / 2 is equivalent to a confidence level of 1 - 0.05 / 2

with(dat, t.test(weight_difference[group == "Treat1"],
weight_difference[group == "Control"],
conf.level = 0.975))
#>
#>  Welch Two Sample t-test
#>
#> data:  weight_difference[group == "Treat1"] and weight_difference[group == "Control"]
#> t = -0.10537, df = 37.985, p-value = 0.9166
#> alternative hypothesis: true difference in means is not equal to 0
#> 97.5 percent confidence interval:
#>  -21.09656  19.27376
#> sample estimates:
#>  mean of x  mean of y
#> -1.8206320 -0.9092341
with(dat, t.test(weight_difference[group == "Treat2"],
weight_difference[group == "Control"],
conf.level = 0.975))
#>
#>  Welch Two Sample t-test
#>
#> data:  weight_difference[group == "Treat2"] and weight_difference[group == "Control"]
#> t = -2.9104, df = 34.572, p-value = 0.006276
#> alternative hypothesis: true difference in means is not equal to 0
#> 97.5 percent confidence interval:
#>  -39.226556  -4.234471
#> sample estimates:
#>   mean of x   mean of y
#> -22.6397477  -0.9092341

################################################################################

# with additional co-variates, you can switch to a regression

dat\$age <- runif(NperTreat*3, 18, 65)

lm1 <- lm(weight_difference ~ group + age, data = dat)
summary(lm1)
#>
#> Call:
#> lm(formula = weight_difference ~ group + age, data = dat)
#>
#> Residuals:
#>     Min      1Q  Median      3Q     Max
#> -50.505 -19.415   2.283  16.157  60.701
#>
#> Coefficients:
#>              Estimate Std. Error t value Pr(>|t|)
#> (Intercept)   1.96136   11.56566   0.170  0.86595
#> groupTreat1  -1.17136    8.02909  -0.146  0.88453
#> groupTreat2 -21.55551    8.00057  -2.694  0.00929 **
#> age          -0.06888    0.24229  -0.284  0.77723
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#>
#> Residual standard error: 25.23 on 56 degrees of freedom
#> Multiple R-squared:  0.1461, Adjusted R-squared:  0.1003
#> F-statistic: 3.193 on 3 and 56 DF,  p-value: 0.03038

# There is at least one significant explanatory variable p = 0.03038
# After accounting for age, there is insufficient evidence to conclude there is a difference due to treatment 1 p = 0.88453
# After accounting for age, there is a significant effect due to Treatment 2 p = 0.00929

• Do you know what to do if logistic regression gives different p-value significance compared to Dunnett's test? Which is preferred? Jan 10 at 14:30
• Unfortunately, it's not a good idea to be in this situation in the first place. The most disciplined approach is to specify the type of test ahead of time and then stick with it. I try to ask myself, which approach most closely matches the type of data and the type of hypotheses that are exhibited by the data and the experiment. Are the there other covariates in the regression besides the treatment? Dunnet's test boils down to a set of contrast statements with the correct p-values. You can create this same thing in a logistic regression. Jan 23 at 18:03
• @R Carnell - Yes, I totally agree. It is the case that logistic regression should be able to replicate dunnett's test, but I think this is only if a stratified logistic regression has the same control as the dunnett's test is used. A standardard logistic regression will be different as the each covariate is adjusted against all other covariates unless there is only one other covariate. Feb 10 at 11:57

There is no reason you cannot just apply ANOVA with a standard regression model in this case. Taking three seperate T-tests and combining them into a single inference is inferior to conducting a single ANOVA with a model that includes data from all the groups. In regard to your concerns about ANOVA, it is important to note that ANOVA is a method of analysis, not a model (see e.g., here). It can be applied to a linear regression model irrespective of the particular terms and interactions you include or exclude from your model.

The simplest way to undertake this kind of analysis with your data would be to form a new variable for the weight difference (i.e., weight at the later time minus weight at the earlier time). You can easily form a linear regression model with the drug as your sole explanatory variable and the weight difference as your response variable. The model formula (in R notation) would be:

weight.diff ~ factor(drug)


If you form a model like this then you can easily apply ANOVA to test whether or not there is a statistical relationship between the drug variable and the weight.diff variable. It is best to do a holistic ANOVA for all groups first, and then proceed down to individual tests afterward, with proper consideration of multiple comparisons. Note also that one thing that is important in this kind of work is for your drug allocations to be randomised so that you are doing an Randomised Controlled Trial (RCT). Randomisation of your drug allocations, and use of placebos for the control group, should ensure that your drug variable is not statistically dependent with any possible confounding factors, which allows you to make causal inferences from your statistical inferences.

• "Taking three seperate T-tests and combining them into a single inference is inferior to conducting a single ANOVA with a model that includes data from all the groups" Also in the case of considering one-sided T-tests? Or maybe two T-tests (since the question about control vs either of the two treatments seems to be more important than the two treatments relative to each other). Apr 1, 2021 at 0:27

Yes, you can compare multiple independent t-tests, but note that by doing so you increase the probability of making a Type-I error. Assuming an $$\alpha$$ of .05, the probability of a Type-I error is 14.3% if comparing between three groups.

Instead of comparing the start and end weights as a pair, compare the difference between the two. I.e, for every participant, subtract the end from the start, and run an ANOVA on that value.

If you compare multiple independent t-test(s) you should counteract the multiple comparison problem.

If you want to go down this path, one way to tackle the multi comparison problem is to use theBonferroni correction but others are available. you can find a bunch of them here.

• I know about the Bonferroni correction (which is used to multiple comparisons). But I did not find any example of multiple paired tests using Bonferroni. I'm using R btw. Mar 29, 2021 at 9:51
• The three t-tests are not independent, so the Bonferroni correction, which assumes they are independent, will be more conservative than it already is. Mar 29, 2021 at 11:27
• Thanks for the comment. Why do you say the three paired t-tests are not independent? How do you know? Mar 29, 2021 at 11:36
• For a quick explanatory introduction read [here and go to "Interpretational Issues"] (spss-tutorials.com/spss-paired-samples-t-test/comment-page-1). For R look at [here] (stats.idre.ucla.edu/r/faq/…). I hope these could be helpful for your work! Mar 29, 2021 at 12:35
• @Numbermind the two sample t-test computes a t-statistic based on two samples (e.g. the placebo and the treatment 1). For these three t-tests, it is not like you are using three times an independent set of two samples. Instead you are using only three samples. Example: if in t-test 1 we find that the control and treatment 1 are close to each other, then the results of t-test 2 (control vs treatment 2) and t-test 3 (treatment 1 vs treatment 2) will be related (and we effectively have only one additional t-test instead of two). Mar 30, 2021 at 14:35

Instead of doing ANOVA, I want to make three independent paired t-tests, i.e, one paired t-test for each one of the groups (measuring weight before and weight after). ... Can I compare the outputs from the three paired t-tests?

You can do this with Tukey's range method

It is actually relatively similar and there are similar rejection regions (similar for rejecting the null hypothesis $$\mu_1=\mu_2=\mu_3$$ Tukey's method versus anova).

The F-statistic in anova can be computed from the t-statistics (when these are computed with the pooled standard deviation).

$$F = \frac{t_1^2+t_2^2+t_3^2}{3}$$

With Tukey's method you are regarding the maximum of the absolute value of the t-statistics.

$$q = max(|t_1|,|t_2|,|t_3|)$$

The method is more complex with ordering of the means and finding groups with significant differences, but the control of the FWER is done based on this maximum range. (While this test is often performed as a post-hoc test for ANOVA, you do not need to do ANOVA before Tukey's method. The FWER is controlled by using the studentized range distribution for the cutoff for $$q$$.)

### Demonstration/visualisation/simulation

Below is a simulation assuming the populations are normal distributed with equal variances and equal means.

We plot two out of the three t-statistics (from $$10^4$$ different simulations). The third t-statistic is fully dependent on these two.

• The t-statistics are correlated and not independent (see the clouds of points being an elongated shape). This is because the different t-statistics are computed with similar groups. If $$t_1$$ relates to the difference 'treatment1 - placebo' and $$t_2$$ relates to the difference 'treatment2 - placebo', then the placebo sample will have a similar effect on both $$t_1$$ and $$t_2$$ which is how they become correlated.

This is the reason why the 'multiple comparison problem' should not be tackled with something like the Šidák correction as suggested by some. Instead you should use the studentized range distribution to determine the FWER.

• For a given maximum range the F-statistic can have different values.

Say the lowest mean is 0 and the highest mean is 1. The total variance will depend on the third mean and is highest when it is close to 0 or 1 and lowest when it is close to 0.5.

So, you might have a situation that the treatment 1 is significantly different according to Tukey's test, but depending on treatment 2 the ANOVA result can be more not significant (and vice-versa: R Tukey HSD Anova: Anova significant, Tukey not?).

Strangely the anova test can fail when the treatment 2 has values further away from the placebo. (E.g. the values 'placebo = 0, treatment1 = 1 and treatment2 = 0' have higher variance than 'placebo = 0, treatment1 = 0.5 and treatment2 = 0'.)

• The rejection regions are both rejecting in 5% of the cases the null hypothesis $$\mu_1=\mu_2=\mu_3$$ if the null hypothesis is correct (type I error). The rejection regions are very similar.

sim <- function(n=20,mu1=0,mu2=0) {
### sampling
x0 <- rnorm(n)
x1 <- rnorm(n, mu1)
x2 <- rnorm(n, mu2)
### compute intermediate statistics
mean_total <- mean(c(x0,x1,x2))
SSR <- sum((x0-mean(x0))^2+
(x1-mean(x1))^2+
(x2-mean(x2))^2)        ### residuals
SSE <- n*((mean(x0)-mean_total)^2+
(mean(x1)-mean_total)^2+
(mean(x2)-mean_total)^2) ### explained
SST <- sum((x0-mean_total)^2+
(x1-mean_total)^2+
(x2-mean_total)^2)      ### total
sig_pooled <- sqrt(SSR/(3*n-3))

### compute test statistics
t1 <- (mean(x1)-mean(x0))/sig_pooled/sqrt(2/n)
t2 <- (mean(x2)-mean(x0))/sig_pooled/sqrt(2/n)
t3 <- (mean(x2)-mean(x1))/sig_pooled/sqrt(2/n)
Fscore <- (SSE/2) / (SSR/(3*n-3))

### output
return(list(t1=t1, t2=t2, t3=t3, Fscore=Fscore))
}

### simulate
set.seed(1)
n = 20
alpha = 0.95
x <- replicate(10^4,sim(n))
t1 <- as.numeric(x[1,])
t2 <- as.numeric(x[2,])
t3 <- as.numeric(x[3,])
Fscore <- as.numeric(x[4,])

### boundaries for colouring
### F-score
f_boundary <- qf(alpha,2,n*3-3)
colf = (Fscore >= f_boundary)
### range
t_boundary <- qtukey(alpha, nmeans = 3, df = n*3-3)/sqrt(2)
#t_boundary <- qt(1-0.5*(1-alpha), df = n*3-3)
colt = 1-(abs(t1) < t_boundary)*(abs(t2) < t_boundary)*(abs(t3) < t_boundary)

### plot results
col = hsv(0.33+colt*0.33-colf*0.33, 1, ((colf+colt)>=1)*0.7,0.5)
plot(t1,t2, col = col, bg = col, pch = 21 , cex = 0.3,
xlab = "t1", ylab = "t2",
ylim = c(-4,4), xlim = c(-4,4))

sum(colf)/length(t1) ### 5.03% outside the F boundary
sum(colt)/length(t1) ### 5.08% outside the Tukey boundary

### F-distribution boundary
### F * 3 = (t1^2+t2^2+t3^2) = (t1^2+t2^2+(t2-t1)^2) =
###                          = (2t1^2 + 2t2^2 - 2t1t2 =  1/2 (t1+t2)^2+ 3/2 (t2-t1)^2
phi <- seq(0,2*pi,0.01)
u <- cos(phi)*sqrt(f_boundary*2)
v <- sin(phi)*sqrt(f_boundary*6)
x1 <- (v-u)/2
x2 <- (u+v)/2
lines(x1,x2,col=2)

### Tukey range boundary
lines(t_boundary*c(1,1,0,-1,-1,0,1),t_boundary*c(0,1,1,0,-1,-1,0), col = 4)

title("comparing anova with Tukey's Method")
legend(-4,4, c("non significant",
"both significant",
"only anova",
"only Tukey's method"), col = c(1,3,2,4), pt.bg = c(1,3,2,4), pch=21, cex = 0.7)


This problem would also be more complex if I needed to add another variable, for example a variable for gender (male and female) to check if gender also affects the treatment.

The most easy would be to formulate this as a linear model and use variance (F-test/anova) or parameter estimates (t-test) to describe the significance. (Effectively these will be the same, anova and t-tests give the same results)

I want to test if drug 1 and drug 2 are effective to reduce weight

If you are more interested in only a few out of all possible comparisons of means, and/or if you are interested in one sided alternative hypotheses ($$H_0: \text{not effective}$$ versus $$H_a: \text{effective and more specifically weight reducing}$$), then you can change the rejection boundaries of the t-test (e.g. use 1 sided t-tests and ignore the 3rd t-statistic for difference between treatments). The result is Dunnett's test, which R Carnell speaks about in their answer, which will be a more powerful test.