# Guideline one using which statistical test?

Are there any guide lines on determining which test to use?

For example, given 100 subjects with both Exam A and Exam B, and some student observations for those exams, i want to compare which exam is harder.

More specifically, I have 100 different subjects (ex. math, french, english, ...etc), and each subjects consists 2 exams, exam A & exam B. Now, I distribute these exams to a class of 100 and if a student score above 50%, then the observation is set as 1 and 0 otherwise.

An example table:

Subject Exam    Class Score
MATH    A           90
MATH    B           10
FRENCH   A           51
FRENCH   B           49
...      ...        ...


Which statistical test should i perform if I want to compare which exam is harder?

Summary of the question (by @user2974951):

• We have 100 students, each student takes test A and B for 100 different subjects. The results of the tests are independent, so the results of test A does not affect the result of test B, and a student score in subject 1 will have no effect on the score of subject 2. The data is already aggregated on the subject and exam level, that is we only have the frequency of students which passed. Our goal is to determine whether test A is harder than test B, based on the aggregated data.
• Don't do that, don't classify the scores into classes. Use raw scores to compute means rather. Are any of the observations dependent? That is, does a student take more than one test? Is a student present in more than one subject? Does a student take both the A and B test? Feb 25, 2021 at 11:44
• @user2974951 unfortunately I am only given this type of data.. the observations are independent. It is true that all 100 students will take both tests for all subjects, but we assume that taking exam A will have no impact when taking exam B, and no relationship between each students. Feb 25, 2021 at 11:54
• If possible, I would analyze data by student, not by class. I would use a multilevel model where student is level 1, class is level 2. This would allow me to assess the degree to which the class predicts scores (i.e. teacher effect) versus student. In R, this could be something like: fit <- lmer(score ~ exam + subject + (exam | class)). See benwhalley.github.io/just-enough-r/fitting-models.html, for examples Feb 25, 2021 at 15:02
• @BrantInman But i want to compare exam A & exam B tho Feb 26, 2021 at 1:09
• I think i should write the question clearer; the column "class score" is actually mentioning the same class with the same 100 students. So there's only one class level and students would be 100 levels Feb 26, 2021 at 1:43

Given the assumption that all observations are independent, that is within the subjects and between the subjects, then a simple t-test can be used to solve this - assuming all the assumptions of the t-test are valid (your data is not really continuous so they might not hold, but it might be a good approximation). Otherwise an equivalent non-parametric test might work better.

Your goal is to determine whether test A is harder than test B, or to put it differently if the frequency of test A is higher than that of test B. Given that you only have aggregated data, the results are interpreted on the frequencies of students which passed, rather than say the average score.

So for ex. you might find that the expected frequency for test A is 67, while the expected frequency for test B is 54. If the standard error is small enough you would find that the difference is statistically significant.

If, however, there is a relation between the exams, then you can use a paired t-test. If there is a relation within the students, then a linear mixed model can be used.

• For the example (...the expected frequency for test A is 67, while the expected frequency for test B is 54...) , what do you mean by the standard error? Of the differences between test A and B, or within test A/B? Mar 3, 2021 at 6:32
• @MathAvengers A t-test would calculate the pooled standard deviation of the means (standard error), assuming equal variances. And it would use this standard error, combined with the mean difference between A and B to construct a confidence interval, and determine whether the result is statistically significant. Mar 3, 2021 at 7:03