# Alternative to an ANOVA

I'm currently attempting to run statistical analysis on data from an experiment but have encountered an issue.

The number of participants is very low (8 in total) and there are 27 conditions after collapsing the data as much as is logically possible.

I run my stats using R statistical package and when attempting to run a Type II or Type III ANOVA the output informs me that the test:

...seems to have failed. Most commonly this is because you have too few subjects
relative to the number of cells in the within-Ss design


I've got around this before by reducing the individual conditions via logical collapsing but as I stated before I've already tried that here.

While a Type I test does work I'm unhappy that there is no assumption tests.

Therefore, can any of you recommend an alternative to an ANOVA I, II, or III that might make sense to run here?

The data consists of 3 Visual Conditions x 9 Auditory Conditions totalling 27 conditions.

I hope I've provided enough information if not I'll be happy to amend.

EDIT:

The design is a 3x9 within-subjects/repeated measures.

The DV is Accuracy of Response.

The IVs are 3x9 visual-audio combinations.

Below is sample data for one participant (subject number 0) from the data frame AGGFRAMED.ANOVA.Collapsed.means:

subject_nr  AudioAll                            Visual1           correct
0           1 Central Beep 1 WN                 Left Circle 1st   0.24
0           Left or Right Beep 1st              Left Circle 1st   0.7
0           Left or Right Beep Opposite 1 WN    Left Circle 1st   0.63
0           Left or Right Beep Same WN          Left Circle 1st   0.63
0           No Beep                             Left Circle 1st   0.48
0           Post Visual Central Beep            Left Circle 1st   0.42
0           Post Visual Left or Right Beep      Left Circle 1st   0.6
0           Pre Visual Left or Right Beep       Left Circle 1st   0.52
0           Two Beeps Same Side                 Left Circle 1st   0.6
0           1 Central Beep 1 WN                 Right Circle 1st  0.38
0           Left or Right Beep 1st              Right Circle 1st  0.78
0           Left or Right Beep Opposite 1 WN    Right Circle 1st  0.76
0           Left or Right Beep Same WN          Right Circle 1st  0.74
0           No Beep                             Right Circle 1st  0.78
0           Post Visual Central Beep            Right Circle 1st  0.52
0           Post Visual Left or Right Beep      Right Circle 1st  0.76
0           Pre Visual Left or Right Beep       Right Circle 1st  0.54
0           Two Beeps Same Side                 Right Circle 1st  0.7
0           1 Central Beep 1 WN                 SIM Circle        0.94
0           Left or Right Beep 1st              SIM Circle        0.1
0           Left or Right Beep Opposite 1 WN    SIM Circle        0.08
0           Left or Right Beep Same WN          SIM Circle        0.39
0           No Beep                             SIM Circle        0.66
0           Post Visual Central Beep            SIM Circle        0.9
0           Post Visual Left or Right Beep      SIM Circle        0.53
0           Pre Visual Left or Right Beep       SIM Circle        0.64
0           Two Beeps Same Side                 SIM Circle        0.46


There are 7 more subjects with the same number of conditions (27).

The code I used to run the ANOVA is as follows:

Exp9CollapsedModel <- ezANOVA(data = AGGFRAMED.ANOVA.Collapsed.means, dv = .(correct), wid = .(subject_nr), within = .(Visual1, AudioAll), type = 3, detailed = TRUE)

Exp9CollapsedModel


The following is the error:

Error in lambda > 0 : invalid comparison with complex values

Error in ezANOVA_main(data = data, dv = dv, wid = wid, within = within, :

The car::Anova() function used to compute results and assumption tests seems to have failed. Most commonly this is because you have too few subjects relative to the number of cells in the within-Ss design. It is possible that trying the ANOVA again with "type=1" may yield results (but definitely no assumption tests).

• You need to tell us more about the design, and the analysis that you ran. A reproducible example would be great, as would the code. – Jeremy Miles Apr 14 '15 at 17:54
• I don't know anything about ezANOVA, not even what package it's in. But I suspect that the model is including interactions of the within-subjects factors, which adds a whole huge number of parameters. You might try a model like lm(correct ~ factor(subject_nr) + AudioAll + Visual1, data = AGGFRAMED.ANOVA.Collapsed.means) and see if it works. You can still use car's Anova function to summarize it. This models subjects as fixed, but if this flies we can talk about better models. – rvl Apr 16 '15 at 21:39
• @rvl Thanks for the response. I've run that code here is an example of the output: '(Intercept) factor(subject_nr)1 0.637454 0.055556 ' I can't seem to format this correctly. I don't know how to interpret this output and it doesn't appear to solve my concenrs about tests of assumptions. – Docconcoct Apr 17 '15 at 11:44
• @Docconcoct -- OK, well, look at summary() of the result. Are there any NA values -- meaning things that can't be estimated? If not, we can try a similar model using the lme4 package that accounts for the random subjects. – rvl Apr 17 '15 at 16:52
• Another question -- is the response a proportion correct or something? Is each based on the same number of trials? – rvl Apr 17 '15 at 16:54

This answer is offered because I don't really know what model ezANOVA is trying to fit. I do know that the ez package uses routines from lme4 and others, so I'm suggesting using those directly rather than via whatever "ez" approaches are actually not so "ez" (for me) to understand...

So here is some code to try, for starters. First, create a variable in your data frame named ntrials, or some such, that has the number of trials underlying each value in correct.

library(lme4)
mod1 = glmer(correct ~ Visual1 + AudioAll + (1| subject_nr),
weights = ntrials, family = binomial,
data = AGGFRAMED.ANOVA.Collapsed.means)
plot(mod1)
summary(mod1)
library(car)
Anova(mod1)


This model fits a mixed model to the logit of the observed proportions, with random subjects. The plot will be the most basic diagnostic plot, and it should look like a random scatter. The summary will show the regression coefficients and tests thereof. The Anova will give a type II-style ANOVA, but with $\chi^2$ tests instead of $F$ tests because of the binomial model.

If the diagnostics look OK, you could do followup comparisons something like this:

library(lsmeans)
lsmeans(mod1, pairwise ~ Visual1, type = "r")
lsmeans(mod1, pairwise ~ AudioAll, type = "r")


These commands obtain marginal averages of the predictions from this model, then transform the results to the original (fraction of successes) scale (the comparisons come out as odds ratios).