Lets say I have a factor called cognitive load that has 3 levels (low, medium, high) and I want to determine how load levels can affect reaction time on some task

There are at least 2 ways that I could conduct this test.

The first would be a one way ANOVA with a single 3 level factor. After running this test I find that the main effect of this factor is not significant. I was always taught that without a significant main effect you shouldn't/dont need to test the differences between each of the groups/levels

The second way is to dummy code the factor and treat it as a regression, predicting reaction time from the 2 generated dummy variables. If I treat low load as my reference group, then my dummy variables would capture the difference between low/medium and low/high. When I conduct this analysis I find that one of my coefficients is significant (e.g. high is significantly different from low load)

So this situation leads to 2 different interpretations of the results. On the one hand I dont have a significant effect of the factor overall, but upon looking at group differences with regression I find something interesting

Which technique should actually be followed? Should I be looking at group differences in ANOVA despite the lack of a significant main effect?

  • $\begingroup$ Read this theanalysisfactor.com/… What are you focusing in your study? If cognitive load ANOVA is sufficient, but if it is the cognitive load levels I will use regression. $\endgroup$ – Robert Aug 12 '16 at 18:23

First, run a histogram of $y$ and look at the dist of $y$ to confirm it's normal. Run the Shapiro-Wilk test on $y$ to test for normality. During ANOVA, run Bartlett's equality of variance test for the three groups, and confirm they don't have different variances. (if they are significantly different and the sample sizes are different, then the Brown and Forsythe W-test is appropriate to use; note that when variances are unequal and samples sizes are equal, ANOVA is supposed to be robust). With the regression approach, you are dipping into the degrees of freedom within the factor. The overall F-test of ANOVA will test that at least one pair of means (every combination of levels) are significantly different -- so you should get something from ANOVA if two levels have significantly different means. The two dummy indicator variables in regression will have coefficients whose values represent each group's change in $y$ from the constant term. So, your regression test will determine if groups 2 and 3 have significantly different $y$-values compared against only the constant group (group 1).

For post-regression diagnostics, you have to also prove that the residuals are normally distributed, so generate a histogram of all the $e_i$'s to confirm this. Look at standardized and leverage residuals, etc., as well.

  • $\begingroup$ It is not the case that if there is heteroskedasticity you must use Kruskal Wallis. Not least one can do a Welch Satterthwaite adjustment with ANOVA just as with t-tests (there are other possibilities). $\endgroup$ – Glen_b Sep 26 '16 at 7:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.