0
$\begingroup$

As the title implies, I am collecting data on student success rates on specific test questions. My department recently enacted a small curriculum change in part of a course's lab segment (intentionally vague), and we wish to evaluate it based on the outcomes it is producing on the final exams.

We give standardized final exams for this course every year and keep (de-identified) records of each student's responses electronically for internal evaluation purposes. I wish to examine the test question on the content affected by the lab curriculum change.

I have data for a semester where a more complex lab activity was used, data when no lab activity was used, and data for the new lab activity. The test question has 5 multiple choice answers, and only one is correct. The test question is virtually the same every year. So, let's assume that the student responses to the questions are all three directly comparable and not affected significantly by any other factors than the lab activity. Given that, let's say that I have the following two sets of data (this is fabricated):

  • On a year without a lab activity, students succeeded at the problem with a rate of 42% (n=309).
  • On a year with the new lab activity, students succeeded at the problem with a rate of 74% (n=205)

What statistical test can I use to compare these two sets of data? I want to be able to determine at some confidence, α, that one year of students answered the question better than the other (or confirm the null hypothesis that the success rates aren't different to a statistically significant amount if that is the case).
If I recall my terminology correctly, this data is binomial in nature. Each student either successfully answered the question or failed to do so (e.g., 1 or 0, true or false).

Improve this question
$\endgroup$
2
  • $\begingroup$ Isn't it just a chi-squared contingency table test, where the rows are without lab and new lab, and the columns success and failure? $\endgroup$
    – Rui Barradas
    Commented Mar 1, 2020 at 10:29
  • $\begingroup$ @RuiBarradas Investigating that possibility, it would appear so. Thanks! I'm going to leave this up for awhile and see if any of our colleagues chime in with more information that might be helpful. $\endgroup$
    – ShayCorgi
    Commented Mar 1, 2020 at 18:24

1 Answer 1

Reset to default
0
$\begingroup$

If you are only looking at one question at a time, and only looking at the correct/incorrect distinction, then you have a $2 \times 2$ contingency table and can use a chi-squared test.

But some other analysis might be more informative:

  • Looking at all the questions at once, I would start out with visualization. And you could get a test of the joint null hypothesis using logistic regression.

  • Maybe look if there is a change in the choice of distractor options?

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.