# How to compare student success rates on test questions?

As the title implies, I am collecting data on student success rates on specific test questions. My department recently enacted a small curriculum change in part of a course's lab segment (intentionally vague), and we wish to evaluate it based on the outcomes it is producing on the final exams.

We give standardized final exams for this course every year and keep (de-identified) records of each student's responses electronically for internal evaluation purposes. I wish to examine the test question on the content affected by the lab curriculum change.

I have data for a semester where a more complex lab activity was used, data when no lab activity was used, and data for the new lab activity. The test question has 5 multiple choice answers, and only one is correct. The test question is virtually the same every year. So, let's assume that the student responses to the questions are all three directly comparable and not affected significantly by any other factors than the lab activity. Given that, let's say that I have the following two sets of data (this is fabricated):

• On a year without a lab activity, students succeeded at the problem with a rate of 42% (n=309).
• On a year with the new lab activity, students succeeded at the problem with a rate of 74% (n=205)

What statistical test can I use to compare these two sets of data? I want to be able to determine at some confidence, α, that one year of students answered the question better than the other (or confirm the null hypothesis that the success rates aren't different to a statistically significant amount if that is the case).
If I recall my terminology correctly, this data is binomial in nature. Each student either successfully answered the question or failed to do so (e.g., 1 or 0, true or false).

• Isn't it just a chi-squared contingency table test, where the rows are without lab and new lab, and the columns success and failure? Mar 1 '20 at 10:29
• @RuiBarradas Investigating that possibility, it would appear so. Thanks! I'm going to leave this up for awhile and see if any of our colleagues chime in with more information that might be helpful. Mar 1 '20 at 18:24

If you are only looking at one question at a time, and only looking at the correct/incorrect distinction, then you have a $$2 \times 2$$ contingency table and can use a chi-squared test.