I have a dataset that looks something like this:
|piece|question|answer |rating| |1 |1 |yes |3.554 | |1 |2 |no |3.554 | |1 |3 |yes |3.554 | ... |2 |1 |no |3.001 | |2 |2 |sometimes|3.001 | |2 |3 |no |3.001 | ... |3 |1 |yes |2.221 | |3 |2 |yes |2.221 | |3 |3 |yes |2.221 |
So 3 yes/sometimes/no questions evaluated for each piece and then a independent rating for that piece. I want to see how good of a job the answers to the questions are at predicting the final rating.
My approach so far has been to run a one-way ANOVA for each question independently using
for question, question_group in questions_answers_ratings.groupby('question'): answer_group_average_ratings =  for answer, answer_group in question_group.groupby('answer'): print question, answer print answer_group["average_rating"].mean() answer_group_average_ratings.append(answer_group["average_rating"].values.tolist()) print question print stats.stats.f_oneway(*answer_group_average_ratings)
Is this a sound approach? I believed the null hypothesis to be true at the outset of this analysis (i.e. that the answers to the questions don't correlate significantly to the rating, so all answers to any question should have similar mean ratings) but ANOVA has shown several questions to have significant p-values.
One concern I have so far is that the answers tend to be distributed very unevenly. So question 4 has a large majority of No's, question 2 has very few sometimes' etc etc.