Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I have a 2x2, between-subjects experimental design (2 independent variables (IVs) with 2 levels each) and one dependent variable (DV). My data are unbalanced and an interaction between the IVs seems likely, so I plan to use ANOVA with Type III Sums of Squares to test whether my DV is different across one of my IVs while controlling for any influence of the other IV (or the interaction, if present).

My question is this: what is the minimum number of data points that I need in each of my 4 cells, to ensure that ANOVA likely won't give spurious results? I'm aware of Simmons et al.'s recommendation (2011) of having at least 20 data point in each cell, but that recommendation only serves to control the test's rate of false negatives. What I'm more worried about is that with a small enough cell size, the statistics on which ANOVA is based probably won't be very reliable, and so the results of the test would be unreliable as well (in terms of either false negatives or false positives). Are there any papers or texts out there that have studied and made recommendations regarding this concern?

share|improve this question
    
How severely unbalanced are the data? –  DoctorAmbient Mar 14 at 18:23
    
    
@DoctorAmbient, I'll let you judge the severity. In one data set, the counts across 4 cells are 16, 17, 13, 30. In another data set, the counts across 4 cells are 13, 10, 8, 14. –  Tyne Mar 19 at 10:45
add comment

1 Answer 1

up vote 8 down vote accepted

There isn't really any absolute minimum except in a trivial sense (if you won't try to test for the interaction, the minimum $n_{ij}$ will be $1$, if you do want to test for interactions, the minimum cell size might be $2$). Instead, there are two issues here:

  1. The first is the question of the robustness of the ANOVA to the violation of assumptions. Like all linear models (regression, $t$-tests, etc.), the ANOVA assumes that the data within each cell (i.e., the residuals) are independent, have equal (homogenous) variance, and are normally distributed. In truth, some of these assumptions are less necessary than others. For instance, with enough data you don't really need the within-cell distributions to be normal. However, what constitutes 'enough data' depends on how far from normal those distributions are. Thus, the further you are from normality, the more data you need. But there is another twist here, namely that with fewer data it is harder (or even impossible) to assess whether the assumptions of the ANOVA are met. So with fewer data per cell, you really are going by blind faith. If the assumptions are not met, then you can have increased type I error rates.

  2. The second issue is the question of power. The probability of getting significance is a function of the size of the effect and the amount of data you have. If the effects are large enough, you will have good power even if you have only 1 datum per cell. I suspect effects that large are uncommon, though. Thus, you need to determine how large of an effect you want to be able to detect with what power (etc.), and calculate your $N$ accordingly.

share|improve this answer
    
Thanks @gung. I avoided mentioning robustness in my post because it seems to be an extremely loose concept. Can you provide (or link to) a detailed description of the relationship between $n_{ij}$ and the deviation of cell $ij$'s distribution from normality? As a rough example, the information that I seek might be represented as a height-map over a 2D plane, where the two dimensions of the plane are $n_{ij}$ and the deviation of cell $ij$'s distribution from normality, and the height from the plane represents the likelihood of ANOVA's type 1 error rate being inflated. –  Tyne Mar 19 at 10:09
    
I don't think robustness is a loose concept, but it is a large topic. There is a great deal of work in statistics on robustness. To understand the robustness of the ANOVA to violations of the assumptions, you will need to specify the exact nature of the violation you are interested in. There are many ways in which a distribution can be 'non-normal', for instance. In general, skew is more damaging that kurtosis, & skews in opposite directions is especially damaging to type I error rates. –  gung Mar 19 at 23:33
    
To clarify, when you say: "the data within each cell (i.e., the residuals) are independent [...]" in your original answer, are you asserting that the data within each cell of the ANOVA are the residuals themselves? That's how I interpret your use of "i.e.", but I've always thought of the residuals as being collected into a single distribution that spans the entire data set (versus four distributions, in my case), which one could then check for normality. Is that wrong? –  Tyne Mar 26 at 19:20
    
(I've read the answer here, but it only asserts that "normality within cells means normality of residuals". I'm concerned with the implication in the other direction. Does normality of residuals mean normality within cells?) –  Tyne Mar 26 at 19:24
    
I suppose the "i.e." is somewhat ambiguous. An ANOVA is (a special case of) a regression model. In a regression model, it is only the residuals that need to be normal (see my answer here). However, in an ANOVA, the predicted value in each cell is the cell mean. Thus, the normality of the residuals = the normality of the data w/i each cell. It is possible to have normality w/i each cell individually, but not normality of all residuals together b/c of heterogeneity, however, if the residuals are normal, all cells will be too. –  gung Mar 26 at 19:44
show 2 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.