8
$\begingroup$

I know this question has been asked in similar ways already, but cannot find a suitable answer to understand it. I have three subsamples defined on programme participation (participants, drop-out, and comparison) and want to test for each of the groups separately whether the difference in means between the groups is significantly different from 0. So, overall I have three tests, mean1 = mean2, mean2 = mean3, mean1 = mean3

I read that using a paired t-test and a regression would result in the same, but that with ANOVA there is a slight difference? Does somebody know more about this and could suggest which one is best suited?

Thanks!

$\endgroup$
3
  • 1
    $\begingroup$ What does the group 'comparison' mean in this context? Also note that ANOVA is a regression, the difference lies in the null-hypotheses generally associated with these methods. $\endgroup$ Commented Apr 17, 2019 at 8:27
  • 1
    $\begingroup$ A paired t-test (also called dependent t-test) is used for dependent samples, which you usually get from repeated measures or otherwise dependent samples. Your situation would call for independent t-tests. However, see Frans Rodenburg's answer why you probably want to first use an ANOVA. $\endgroup$
    – morphist
    Commented Apr 17, 2019 at 10:18
  • $\begingroup$ See stats.stackexchange.com/questions/268006/… $\endgroup$ Commented Dec 10, 2019 at 10:57

3 Answers 3

20
$\begingroup$

ANOVA vs $t$-tests

With ANOVA, you generally first perform an omnibus test. This is a test against the null-hypothesis that all group means are equal ($\mu_1=\mu_2=\mu_3$).

Only if there is sufficient evidence against this hypothesis, a post-hoc analysis can be run which is very similar to using 3 pairwise $t$-tests to check for individual differences. The most commonly used is called Tukey's Honest Significant Difference (or Tukey's HSD) and it has two important differences with a series of $t$-tests:

  • It uses the studentized range distribution instead of the $t$-distribution for $p$-values / confidence intervals;
  • It corrects for multiple testing by default.

The latter is the important part: Since you are testing three hypotheses, you have an inflated chance of at least one false positive. Multiple testing correction can also be applied to three $t$-tests, but with the ANOVA + Tukey's HSD, this is done by default.

A third difference with separate $t$-tests is that you use all your data, not group per group. This can be advantageous, as it allows for easier diagnostics of the residuals. However, it also means you may have to resort to alternatives to the standard ANOVA in case variances are not approximately equal among groups, or another assumption is violated.

ANOVA vs Linear Regression

ANOVA is a linear regression with only additions to the intercept, no 'slopes' in the colloquial sense of the word. However, when you use linear regression with dummy variables for each of your three categories, you will achieve identical results in terms of parameter estimates.

The difference is in the hypotheses you would usually test with a linear regression. Remember, in ANOVA, the tests are: omnibus, then pairwise comparisons. In linear regression you usually test whether:

  • $\beta_0 = 0$, testing whether the intercept is significantly non-zero;
  • $\beta_j = 0$, where $j$ is each of your variables.

In case you only have one variable (group), one of its categories will become the intercept (i.e., the reference group). In that case, the tests performed by most statistical software will be:

  • Is the estimate for the reference group significantly non-zero?
  • Is the estimate for $(\text{group 1}) - (\text{reference group})$ significantly non-zero?
  • Is the estimate for $(\text{group 2}) - (\text{reference group})$ significantly non-zero?

This is nice if you have a clear reference group, because you can then simply ignore the (usually meaningless) intercept $p$-value and only correct the other two for multiple testing. This saves you some power, because you only correct for two tests instead of three.

So to summarize, if the group you call comparison is actually a control group, you might want to use linear regression instead of ANOVA. However, the three tests you say you want to do in your question resemble that of an ANOVA post-hoc or three pairwise $t$-tests.

$\endgroup$
3
  • $\begingroup$ It could be added than in linear regression, you can also test differences between any two groups (H0: beta_i=beta_j), although that is not a default option in most statistical packages. $\endgroup$
    – Pere
    Commented Apr 17, 2019 at 18:48
  • $\begingroup$ Thank you for the great explanation! $\endgroup$
    – Papayapap
    Commented May 8, 2019 at 10:04
  • $\begingroup$ Great explanation, indeed! What would happen in a ANOVA vs a Mixed effect model ? $\endgroup$
    – Rosa Maria
    Commented Apr 2, 2022 at 19:09
1
$\begingroup$

I think another way of looking at the differences is that the regression model allows you to factor in other confounding factors while evaluating the treatment effect and testing for statistical significance while the t-test/ANOVA is simply evaluating the statistical significance assuming that there are no confounding factors (i.e. if we look at the distribution of the confounders between treatment vs. control vs other groups, the distributions are exactly the same). In general, I think the regression approach is the best but more complex to interpret and communicate to people.

ANOVA vs. t-test is just the difference between testing between two groups (e.g. control vs. treatment ) vs. more than two groups (control vs. treatment 1 vs. treatment 2...) ANOVA just tests the variance between groups vs. within the group and tests the hypothesis that there is one group that is statistically significantly different than other groups.

$\endgroup$
0
$\begingroup$

paired t-test is only used when you have two groups. The name already says about the context in which it should be used. You should use ANOVA in this particular situation when you have more than two groups in the grouping variable.

$\endgroup$
5
  • $\begingroup$ Why should we use ANOVA and not a linear regression? $\endgroup$
    – Papayapap
    Commented Apr 17, 2019 at 8:51
  • $\begingroup$ Result won't change in either case, if you do not add any additional variables in the regression. However, interpretation will be different. $\endgroup$
    – Ahmed Arif
    Commented Apr 18, 2019 at 9:18
  • 2
    $\begingroup$ Usually people use the phrase 'paired t-test' when the individual data points can be paired up between the samples. That doesn't seem to be apparent in the original question. $\endgroup$ Commented Apr 18, 2019 at 12:23
  • $\begingroup$ @AhmedArif Is it true to say that if you add other potentially confounding variables in the regression, the treatment effect using the regression approach would be more accurate since it accounts for those potential bias factors ? Vs. t-tests dont have that ability to account for the confounding variables and only give the treatment effect assuming the distribution between treatment and control relative to the confounding factors are exactly the same? Thanks! $\endgroup$
    – SriK
    Commented Sep 23, 2021 at 22:16
  • $\begingroup$ Yes, this is true. T test or ANOVA are not able to account for any other potential confounding variables. $\endgroup$
    – Ahmed Arif
    Commented Mar 11, 2022 at 23:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.