8
$\begingroup$

I know this question has been asked in similar ways already, but cannot find a suitable answer to understand it. I have three subsamples defined on programme participation (participants, drop-out, and comparison) and want to test for each of the groups separately whether the difference in means between the groups is significantly different from 0. So, overall I have three tests, mean1 = mean2, mean2 = mean3, mean1 = mean3

I read that using a paired t-test and a regression would result in the same, but that with ANOVA there is a slight difference? Does somebody know more about this and could suggest which one is best suited?

Thanks!

$\endgroup$
3
  • 1
    $\begingroup$ What does the group 'comparison' mean in this context? Also note that ANOVA is a regression, the difference lies in the null-hypotheses generally associated with these methods. $\endgroup$ – Frans Rodenburg Apr 17 '19 at 8:27
  • 1
    $\begingroup$ A paired t-test (also called dependent t-test) is used for dependent samples, which you usually get from repeated measures or otherwise dependent samples. Your situation would call for independent t-tests. However, see Frans Rodenburg's answer why you probably want to first use an ANOVA. $\endgroup$ – morphist Apr 17 '19 at 10:18
  • $\begingroup$ See stats.stackexchange.com/questions/268006/… $\endgroup$ – kjetil b halvorsen Dec 10 '19 at 10:57
16
$\begingroup$

ANOVA vs $t$-tests

With ANOVA, you generally first perform an omnibus test. This is a test against the null-hypothesis that all group means are equal ($\mu_1=\mu_2=\mu_3$).

Only if there is sufficient evidence against this hypothesis, a post-hoc analysis can be run which is very similar to using 3 pairwise $t$-tests to check for individual differences. The most commonly used is called Tukey's Honest Significant Difference (or Tukey's HSD) and it has two important differences with a series of $t$-tests:

  • It uses the studentized range distribution instead of the $t$-distribution for $p$-values / confidence intervals;
  • It corrects for multiple testing by default.

The latter is the important part: Since you are testing three hypotheses, you have an inflated chance of at least one false positive. Multiple testing correction can also be applied to three $t$-tests, but with the ANOVA + Tukey's HSD, this is done by default.

A third difference with separate $t$-tests is that you use all your data, not group per group. This can be advantageous, as it allows for easier diagnostics of the residuals. However, it also means you may have to resort to alternatives to the standard ANOVA in case variances are not approximately equal among groups, or another assumption is violated.

ANOVA vs Linear Regression

ANOVA is a linear regression with only additions to the intercept, no 'slopes' in the colloquial sense of the word. However, when you use linear regression with dummy variables for each of your three categories, you will achieve identical results in terms of parameter estimates.

The difference is in the hypotheses you would usually test with a linear regression. Remember, in ANOVA, the tests are: omnibus, then pairwise comparisons. In linear regression you usually test whether:

  • $\beta_0 = 0$, testing whether the intercept is significantly non-zero;
  • $\beta_j = 0$, where $j$ is each of your variables.

In case you only have one variable (group), one of its categories will become the intercept (i.e., the reference group). In that case, the tests performed by most statistical software will be:

  • Is the estimate for the reference group significantly non-zero?
  • Is the estimate for $(\text{group 1}) - (\text{reference group})$ significantly non-zero?
  • Is the estimate for $(\text{group 2}) - (\text{reference group})$ significantly non-zero?

This is nice if you have a clear reference group, because you can then simply ignore the (usually meaningless) intercept $p$-value and only correct the other two for multiple testing. This saves you some power, because you only correct for two tests instead of three.

So to summarize, if the group you call comparison is actually a control group, you might want to use linear regression instead of ANOVA. However, the three tests you say you want to do in your question resemble that of an ANOVA post-hoc or three pairwise $t$-tests.

$\endgroup$
2
  • $\begingroup$ It could be added than in linear regression, you can also test differences between any two groups (H0: beta_i=beta_j), although that is not a default option in most statistical packages. $\endgroup$ – Pere Apr 17 '19 at 18:48
  • $\begingroup$ Thank you for the great explanation! $\endgroup$ – Papayapap May 8 '19 at 10:04
0
$\begingroup$

paired t-test is only used when you have two groups. The name already says about the context in which it should be used. You should use ANOVA in this particular situation when you have more than two groups in the grouping variable.

$\endgroup$
3
  • $\begingroup$ Why should we use ANOVA and not a linear regression? $\endgroup$ – Papayapap Apr 17 '19 at 8:51
  • $\begingroup$ Result won't change in either case, if you do not add any additional variables in the regression. However, interpretation will be different. $\endgroup$ – Ahmed Arif Apr 18 '19 at 9:18
  • 1
    $\begingroup$ Usually people use the phrase 'paired t-test' when the individual data points can be paired up between the samples. That doesn't seem to be apparent in the original question. $\endgroup$ – svenhalvorson Apr 18 '19 at 12:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.