5
$\begingroup$

I've an naive question, I haven't a statistician background so sorry if I can't express myself.

Imagine this situation, I have several sample groups (control, treatment 1, treatment 2, ...). I would like to know if it is better to perform a multiple t-test or ANOVA with a post test (like Bonferroni). I know that if I use multiple t-test it will increase the chance of type I error.

Another question, what is the main difference between ANOVA followed by a post test and multiple t-test followed by a post test like BH procedure?

What are the advantages of the first in comparison to multiple t-test followed by p-value correction?

$\endgroup$

1 Answer 1

4
$\begingroup$

I would perform a regression if I were you. With regressions you not only get a significance test, you also get an estimate of the differences between groups. You don't get this with an ANOVA. You can code the treatments as several dummy-coded variables. If you have four treatments, control, treatment1, treatment2, treatment3, then you need three dummy variables. These are binary variables coded 0 or 1 depending on whether the observation in question belongs to the dummy group. It's easiest to explain if you can see the data. Here's some fake data. See how the treatment1_dummy variable is coded 1 if the observation falls in the same row as treatment1 in the group variable and 0 for all the other observations, treatment2_dummy is coded 1 if the observation belongs falls in the same row as treatment2 in the group variable and so on.

       outcome      group treatment1_dummy treatment2_dummy treatment3_dummy
1  -0.17792783    control                0                0                0
2   1.36817934    control                0                0                0
3  -0.04472688    control                0                0                0
4  -1.07506861    control                0                0                0
5  -0.18948026    control                0                0                0
6   4.73309782 treatment1                1                0                0
7   4.23008616 treatment1                1                0                0
8   3.98634961 treatment1                1                0                0
9   5.07688582 treatment1                1                0                0
10  6.04681029 treatment1                1                0                0
11  9.89827005 treatment2                0                1                0
12 10.01896885 treatment2                0                1                0
13 10.63345666 treatment2                0                1                0
14  9.86221136 treatment2                0                1                0
15 10.23407764 treatment2                0                1                0
16 13.52274846 treatment3                0                0                1
17 17.07765091 treatment3                0                0                1
18 15.17727922 treatment3                0                0                1
19 15.17477509 treatment3                0                0                1
20 16.60549137 treatment3                0                0                1 

I created the data so the control group's mean outcome value is around 0, the treatment1 group's mean is about 5, treatment2 group's is around 10, and treatment3 is around 15. The means would look like this

enter image description here

Now you just do a simple linear regression, entering the dummy variables in as predictors. I'm doing this in R but this method will be possible in all software packages.

mod <- lm(outcome ~ treatment1_dummy + treatment2_dummy + treatment3_dummy, data = df)

summary(mod)

# Output
# Coefficients:
#                 Estimate Std. Error t value Pr(>|t|)    
# (Intercept)       0.06526    0.44090   0.148    0.884    
# treatment1_dummy  4.55927    0.62353   7.312 1.74e-06 ***
# treatment2_dummy 10.73874    0.62353  17.223 9.46e-12 ***
# treatment3_dummy 14.57315    0.62353  23.372 8.55e-14 ***

These are uncorrected for multiple comparisons. Because there is no dummy variable for the control group, the coefficients in the output table above represent the estimated difference in average score between the treatment group in question and the reference group (i.e. the control group). The '(Intercept)' coefficient tells you the estimated score in the control group (i.e. unlike the other coefficients this is just the estimated average score in that group, not a test of between-group difference in average score).

Bonferroni is fine for error correction but, like ANOVA, a little old-fashioned. It can be overly conservative and, when there are lots of comparisons, end up correcting out legitimate effects! A really good method is the Benjamini-Hochberg procedure (also known as the false discovery rate, or fdr). Not sure whether SPSS has it as a default but would be surprised if there were no SPSS macros out there that can help you do it.

Here is the output from all pairwise group comparisons, controlled using the false discovery rate method

Linear Hypotheses:
                             Estimate Std. Error t value Pr(>|t|)    
treatment1 - control == 0      4.5593     0.6235   7.312 2.09e-06 ***
treatment2 - control == 0     10.7387     0.6235  17.223 2.84e-11 ***
treatment3 - control == 0     14.5731     0.6235  23.372 5.13e-13 ***
treatment2 - treatment1 == 0   6.1795     0.6235   9.911 4.67e-08 ***
treatment3 - treatment1 == 0  10.0139     0.6235  16.060 5.47e-11 ***
treatment3 - treatment2 == 0   3.8344     0.6235   6.150 1.40e-05 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
(Adjusted p values reported -- fdr method)

Notice the first three comparisons are the same as in the first output. The error correction hasn't changed the p-values much because they were so significant already. But this is a neat way to get what you want, obtain significance tests as well as estimates of unstandardised effect size, and control for multiple comparisons.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.