0
$\begingroup$

I work for a school. I have two sets of data. Do I need to compare an apple with an orange?

Set 1: test scores from our school. Min score is 0. Max score is 100.

Set 2: test scores from third-party exams outside school. Min score is 100. Max score is 500.

I try to use data from Set 2 as a baseline to see the progress of students. First, I convert Set 1 and 2 to z scores and then compare them. For example, Peter scores 70 in Science and this is converted to a z score of 1.1. He also scores 350 in the external exam and this score is converted to 1.5.

The third-party exam was conducted in 2015. School test is conducted in 2017. Since 1.5 is greater than 1.1, maybe I can say his Science has improved since 2015?

However, there is a problem. Let's say Peter has done really well in external exams, his z score may become 3.0. He is also a top student in our school and he has received a score of 95 on our first school test. This 95 may be converted to 2.7. Since 2.7 - 3.0 is -0.3, his growth is negative. This is not right. Because the two populations have different means, the z scores will be different. Even if Peter has done so well in all school tests, his z scores of all of them may not reach 3.0. Therefore his growth will always be negative.

My question is: we have 2 sets of data with the same population but the nature of the data is not the same. How do I perform a meaningful comparison between them in statistics?

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ The example you provide of a student who does well on an initial exam who then does worse on follow-up exams is virtually identical to the textbook example of regression towards the mean. Indeed the Wikipedia page uses exam results as an example as well. You certainly need to be careful in such cases. I would suggest carefull reading up on how to avoid regression towards the mean. As mentioned by @user3903581 below, comparing quantiles rather than z-scores will work better here. $\endgroup$
    – AaronDefazio
    Mar 31, 2017 at 3:30
  • $\begingroup$ @AaronDefazio. Thank you for the link. I will do some research on "regression towards the mean". $\endgroup$
    – Stephen King
    Mar 31, 2017 at 4:12

1 Answer 1

Reset to default
2
$\begingroup$

If the populations are the exact same populations, you can convert grades from both tests to ranks and compare relative rankings. I think that with the sort of data you have, all you can hope to do is compare the relative standing of the students within the population. In order to assess absolute growth, it is necessary to have data regarding a reference group of students (for example, from the entire school district).

Improve this answer
$\endgroup$
1
  • $\begingroup$ It is a good idea to use rank. But unfortunately students can change subjects. Peter may did the Science test in 2015 and he drops Science and take up Geography in 2017. It works if we rank students for the whole cohort. It does not work if we rank them by class and, worst, they can change subjects during school years. $\endgroup$
    – Stephen King
    Mar 31, 2017 at 4:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.