Instead of doing ANOVA, I want to make three independent paired t-tests, i.e, one paired t-test for each one of the groups (measuring weight before and weight after).
...
Can I compare the outputs from the three paired t-tests?
You can do this with Tukey's range method
It is actually relatively similar and there are similar rejection regions (similar for rejecting the null hypothesis $\mu_1=\mu_2=\mu_3$ Tukey's method versus anova).
The F-statistic in anova can be computed from the t-statistics (when these are computed with the pooled standard deviation).
$$F = \frac{t_1^2+t_2^2+t_3^2}{3}$$
With Tukey's method you are regarding the maximum of the absolute value of the t-statistics.
$$q = max(|t_1|,|t_2|,|t_3|)$$
The method is more complex with ordering of the means and finding groups with significant differences, but the control of the FWER is done based on this maximum range. (While this test is often performed as a post-hoc test for ANOVA, you do not need to do ANOVA before Tukey's method. The FWER is controlled by using the studentized range distribution for the cutoff for $q$.)
Demonstration/visualisation/simulation
Below is a simulation assuming the populations are normal distributed with equal variances and equal means.
We plot two out of the three t-statistics (from $10^4$ different simulations). The third t-statistic is fully dependent on these two.
The t-statistics are correlated and not independent (see the clouds of points being an elongated shape). This is because the different t-statistics are computed with similar groups. If $t_1$ relates to the difference 'treatment1 - placebo' and $t_2$ relates to the difference 'treatment2 - placebo', then the placebo sample will have a similar effect on both $t_1$ and $t_2$ which is how they become correlated.
This is the reason why the 'multiple comparison problem' should not be tackled with something like the Šidák correction as suggested by some. Instead you should use the studentized range distribution to determine the FWER.
For a given maximum range the F-statistic can have different values.
Say the lowest mean is 0 and the highest mean is 1. The total variance will depend on the third mean and is highest when it is close to 0 or 1 and lowest when it is close to 0.5.
So, you might have a situation that the treatment 1 is significantly different according to Tukey's test, but depending on treatment 2 the ANOVA result can be more not significant (and vice-versa: R Tukey HSD Anova: Anova significant, Tukey not?).
Strangely the anova test can fail when the treatment 2 has values further away from the placebo. (E.g. the values 'placebo = 0, treatment1 = 1 and treatment2 = 0' have higher variance than 'placebo = 0, treatment1 = 0.5 and treatment2 = 0'.)
The rejection regions are both rejecting in 5% of the cases the null hypothesis $\mu_1=\mu_2=\mu_3$ if the null hypothesis is correct (type I error). The rejection regions are very similar.
sim <- function(n=20,mu1=0,mu2=0) {
### sampling
x0 <- rnorm(n)
x1 <- rnorm(n, mu1)
x2 <- rnorm(n, mu2)
### compute intermediate statistics
mean_total <- mean(c(x0,x1,x2))
SSR <- sum((x0-mean(x0))^2+
(x1-mean(x1))^2+
(x2-mean(x2))^2) ### residuals
SSE <- n*((mean(x0)-mean_total)^2+
(mean(x1)-mean_total)^2+
(mean(x2)-mean_total)^2) ### explained
SST <- sum((x0-mean_total)^2+
(x1-mean_total)^2+
(x2-mean_total)^2) ### total
sig_pooled <- sqrt(SSR/(3*n-3))
### compute test statistics
t1 <- (mean(x1)-mean(x0))/sig_pooled/sqrt(2/n)
t2 <- (mean(x2)-mean(x0))/sig_pooled/sqrt(2/n)
t3 <- (mean(x2)-mean(x1))/sig_pooled/sqrt(2/n)
Fscore <- (SSE/2) / (SSR/(3*n-3))
### output
return(list(t1=t1, t2=t2, t3=t3, Fscore=Fscore))
}
### simulate
set.seed(1)
n = 20
alpha = 0.95
x <- replicate(10^4,sim(n))
t1 <- as.numeric(x[1,])
t2 <- as.numeric(x[2,])
t3 <- as.numeric(x[3,])
Fscore <- as.numeric(x[4,])
### boundaries for colouring
### F-score
f_boundary <- qf(alpha,2,n*3-3)
colf = (Fscore >= f_boundary)
### range
t_boundary <- qtukey(alpha, nmeans = 3, df = n*3-3)/sqrt(2)
#t_boundary <- qt(1-0.5*(1-alpha), df = n*3-3)
colt = 1-(abs(t1) < t_boundary)*(abs(t2) < t_boundary)*(abs(t3) < t_boundary)
### plot results
col = hsv(0.33+colt*0.33-colf*0.33, 1, ((colf+colt)>=1)*0.7,0.5)
plot(t1,t2, col = col, bg = col, pch = 21 , cex = 0.3,
xlab = "t1", ylab = "t2",
ylim = c(-4,4), xlim = c(-4,4))
sum(colf)/length(t1) ### 5.03% outside the F boundary
sum(colt)/length(t1) ### 5.08% outside the Tukey boundary
### F-distribution boundary
### F * 3 = (t1^2+t2^2+t3^2) = (t1^2+t2^2+(t2-t1)^2) =
### = (2t1^2 + 2t2^2 - 2t1t2 = 1/2 (t1+t2)^2+ 3/2 (t2-t1)^2
phi <- seq(0,2*pi,0.01)
u <- cos(phi)*sqrt(f_boundary*2)
v <- sin(phi)*sqrt(f_boundary*6)
x1 <- (v-u)/2
x2 <- (u+v)/2
lines(x1,x2,col=2)
### Tukey range boundary
lines(t_boundary*c(1,1,0,-1,-1,0,1),t_boundary*c(0,1,1,0,-1,-1,0), col = 4)
title("comparing anova with Tukey's Method")
legend(-4,4, c("non significant",
"both significant",
"only anova",
"only Tukey's method"), col = c(1,3,2,4), pt.bg = c(1,3,2,4), pch=21, cex = 0.7)
This problem would also be more complex if I needed to add another variable, for example a variable for gender (male and female) to check if gender also affects the treatment.
The most easy would be to formulate this as a linear model and use variance (F-test/anova) or parameter estimates (t-test) to describe the significance. (Effectively these will be the same, anova and t-tests give the same results)
I want to test if drug 1 and drug 2 are effective to reduce weight
If you are more interested in only a few out of all possible comparisons of means, and/or if you are interested in one sided alternative hypotheses ($H_0: \text{not effective}$ versus $H_a: \text{effective and more specifically weight reducing}$), then you can change the rejection boundaries of the t-test (e.g. use 1 sided t-tests and ignore the 3rd t-statistic for difference between treatments). The result is Dunnett's test, which R Carnell speaks about in their answer, which will be a more powerful test.
