I have a dataset that looks something like this:

|piece|question|answer   |rating|
|1    |1       |yes      |3.554 |
|1    |2       |no       |3.554 |
|1    |3       |yes      |3.554 |
|2    |1       |no       |3.001 |
|2    |2       |sometimes|3.001 |
|2    |3       |no       |3.001 |
|3    |1       |yes      |2.221 |
|3    |2       |yes      |2.221 |
|3    |3       |yes      |2.221 |

So 3 yes/sometimes/no questions evaluated for each piece and then a independent rating for that piece. I want to see how good of a job the answers to the questions are at predicting the final rating.

My approach so far has been to run a one-way ANOVA for each question independently using scipy.stats.f_oneway:

for question, question_group in questions_answers_ratings.groupby('question'):
  answer_group_average_ratings = []
  for answer, answer_group in question_group.groupby('answer'):
    print question, answer
    print answer_group["average_rating"].mean()

  print question
  print stats.stats.f_oneway(*answer_group_average_ratings)

Is this a sound approach? I believed the null hypothesis to be true at the outset of this analysis (i.e. that the answers to the questions don't correlate significantly to the rating, so all answers to any question should have similar mean ratings) but ANOVA has shown several questions to have significant p-values.

One concern I have so far is that the answers tend to be distributed very unevenly. So question 4 has a large majority of No's, question 2 has very few sometimes' etc etc.

New contributor
Chris is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
  • $\begingroup$ What is a piece, what does it represent? Are you interested only in the global predicting power? $\endgroup$ – user2974951 Feb 12 at 8:22
  • $\begingroup$ It's a physical product. My overall goal is to try and decide if the questions are worth the time they take to answer. If the rating is completely unrelated to the answers to the questions I want to stop the process of asking them. $\endgroup$ – Chris Feb 13 at 19:06

An ANOVA seems reasonable given your data. I am assuming that questions 1, 2 and 3 (and so on) are the same across all pieces and that the pieces are unrelated. ANOVA is quite robust to unequal sample sizes, so if you get significant results then there is a good chance that there are in fact differences.

Also, I see in your code that you used a oneway ANOVA, but as I see it you have two independent variables, Question and Answer, so how does that work?


Your Answer

Chris is a new contributor. Be nice, and check out our Code of Conduct.

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.