ANOVA - independence assumption

I have conducted a study in which I have gathered 3 sentences per participant. These sentences were then classified and the words per sentence were counted. I want to test if there are significant differences in the word count means for the different categories of the classification.

However, because the sentences are not fully independent (each participant has entered 3 sentences), I am worried that ANOVA would not be the right analysis, since it assumes the independence of the cases.

Would it be better to carry out non-parametric statistics on transformed word counts, or is it okay to do an ANOVA?

PS I have asked a related question about using the $\chi^2$ test in a similar situation to identify interactions between factors.

-
Could you elaborate a little bit more on the "different categories of the classification"? I'm thinking a multi-level model might be useful here... –  dominic999 Apr 14 '12 at 23:18
Each sentence was classified using 3 classifications: type of sentence (nominal: {phrase, question, keywords}), data set (nominal, 3 choices, selected by participant), and distance to data set (6 levels, ordinal). –  Lars Grammel Apr 15 '12 at 0:20
add comment

1 Answer

In this case, unlike your $\chi^2$ question, the unit of analysis is a sentence and the randomness is very much associated with the sentence level, and hence you do need to worry about their independence. As @dominic999 suggests, a multi-level model (also known as a mixed effects model) will probably be appropriate, with participant as a grouping level and number of words as the response. Because the response is a count, you will probably find a mixed effects generalized linear model with a Poisson distribution is what you want.

However, an additional problem is that your classification happens after the experiment and hence is likely to be conflated with your response. Each time someone classifies the sentence they may be influenced by how many words the sentence has. This makes the experiment very different from a design where the classification is an independent variable in which you are interested in its impact on the response. There might be a way around this with some kind of two-way causality model similar to structural equation modelling; but my hunch is that better would be to take a step back and think through the aims of the research. Depending on more information about how your classification works, this issue may or may not be a problem.

-
Thanks! The study is an exploratory study to get an initial understanding of the nature of those sentences (unlike an experiment, I had no initial hypotheses and treatments). I want to quantify the qualitative findings I had and to inform about potential interactions between the classifications, but I do not attempt to establish any cause-effect relationship. The categories are whether the sentences are {questions,phrases,keywords}, what data set they used when formulating them (3 data sets, nominal scale), and how close they are to the data set (6 levels, ordinal scale). –  Lars Grammel Apr 15 '12 at 0:19
add comment