I have conducted an experimental study, with 1 within-variable (time: T1 and T2) and 1 between-variable (group: control and treatment), measuring just one dependent variable.

I understand that this is a design which would require a mixed ANOVA analysis. Of course two distinct t-tests would be much easier: one dependent and one independent two-sample-t-tests.

What is problematic about using two distinct t-tests in comparison to mixed ANOVA (despite ignoring the interaction effect, which I assume to not exist)?

Thanks for advice.

Update: What I've done so far (in R) is:

t.test(Con$DELTA, mu=0, alternative = c("greater"))

t.test(Exp$DELTA, mu=0, alternative = c("greater"))

Two single dependent/paired t.tests for each group, one-sided, because I'm just interested in each groups behavior change success. Afterwards I can compare both groups (with independent two-sample-t-test):

t.test(Con$DELTA, Exp$DELTA)

two-sided test, because I'm not sure which group is better than the other. Each t-test represents one stand-alone hypothesis.

  • 2
    The 2-way between-by-within ANOVA (or mixed effect ANOVA) will allow you to test for an interaction. Further, if there is no significant interaction, then the 2 main effects will have more power than their individual t-test equivalents because you use all of the data to estimate your variance components. Although this assumes you have homogeneity of variance. – Moose Apr 6 '16 at 12:30
  • Ok, but despite giving up some power, it's not forbidden to test two hypotheses with seperate t-tests? In my experimental setting there will not be any interaction effect I think, or at least has no meaning. The groups are independent. – Mac Apr 6 '16 at 14:13
  • 2
    Well, assuming that T1 is before treatment and T2 is after treatment, you should hope for an interaction. The usual interaction in this case is that treatment has no effect at T1 (the group means are equal) because patients were randomised, but then at T2, after treatment, there is a difference between the groups. – Moose Apr 7 '16 at 11:02
  • 1
    Look into using a repeated 2x2 ANOVA or an ANCOVA. Running different t-tests is not the way to go. – Moose Apr 7 '16 at 11:12

A practical situation in which the t-test approach is inferior to mixed models in this simple setting is when you have missing data. In this case, the t-test approach is only valid under the missing completely at random assumption, whereas the mixed models are valid under both the missing completely at random and the missing at random assumptions.

The independent samples t-test on the change scores is equivalent to testing the interaction between the between and within factors in a mixed ANOVA. The p-value will be identical (assuming you use the same homogeneity of variance assumptions on both). This test asks whether the change scores differ between treatment and control, which is clearly your question of primary concern.

The repeated measures t-tests within each condition of the between-subjects factor correspond to simple effects tests for the within-subjects factor at each level of the between factor. In this case, the p-values will differ based on whether you performed these tests in the mixed ANOVA or t-test frameworks because the variance used in the denominator of the corresponding test-statistics will differ. The variance in the mixed ANOVA uses the variance from the entire sample, and this is more accurate and has more degrees of freedom. The variance in the t-test framework only considers the variance in each group of the between-subjects factor, which estimates the population variance poorly relative to the whole sample estimate.

Note that none of the t-tests correspond to main effects, but main effects are clearly not of interest here. You want to know if the between-subjects factor affected the change scores of the individuals. This is an inherently interaction-based research question. The independent samples t-test answers this question completely, and the repeated measures t-tests are probes to determine whether the changes in each of level of the between-subjects factor were meaningful in themselves.

One way to estimate all the quantities of interest using the correct variance estimates would be the following:

Create a binary condition variable TC (treatment/control) and a change score variable delta. The run a regression of delta against TC. The coefficient on TC is the treatment effect on the change scores. The intercept is the mean of the reference category (which is probably the control), and its test statistic corresponds to the test of whether the change score in the control condition is different from 0. You can then change the labels of treatment and control in the TC variable, run the regression again, and now the intercept will be the treatment group mean change score, which again you can test if different from 0. This procedure does not allow you to correct for alpha inflation in the simple effects tests.

Since you are measuring two time points you need a repeated measure ANOVA. A t-test cannot control for time.

  • 2
    I thought dependent, repeated or within are used synonymous? I don't have a profound proof, but compare e.g. this site: statistics.laerd.com/statistical-guides/… information: A dependent t-test is an example of a "within-subjects" or "repeated-measures" statistical test. regards – Mac Apr 6 '16 at 9:10

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.