I am looking to verify if a method that was used to predict a test score based on others' test scores is valid. The scenario is as follows:

John was entitled to take a promotional exam at work that he missed. John was given a different test (Exam 2) than the one he missed (Exam 1), as it was more current and relevant, however his score would be adjusted to an equivalent score of the test takers who took the original exam (Exam 1).

John takes exam 2 and scores a 75. John's score is then differenced with the average score, and then divided by the standard deviation of all the scores for exam 2. That is then converted to a percentile.

That percentile is then used to determine what score John would have obtained on exam 1, that was given to a completely different group of people.

For example, John scores in the 80th percentile on exam 2, so he is given a score that is equivalent to the 80th percentile on exam 1.

I don't think this is a valid method or based on any solid methodology. The pools of test takers are different from test to test, so how could there be any controlled variables. My analogy is that a group of private school kids with the best education available takes an SAT and scores a 1400, in the 70th percentile. You then cannot say with any degree of certainty or accuracy that if you compare that to a group of lower income public school students, that he score would also be in the 70th percentile, which could possibly be a score of 750. This doesn't make sense how this can be used to calculate an equivalent score.

This is an ACTUAL situation for a civil service promotional exam that was given, that I am trying to fight.

| cite | improve this question | | | | |
  • 2
    $\begingroup$ Perhaps this is more of a question for a lawyer than a statistician, considering. $\endgroup$ – Kodiologist Feb 6 '17 at 21:50

I think the biggest concern with the procedure you describe is the implied equivalence of exam 1 and 2. Given your statement that exam 2 is more current and relevant, what evidence is there to justify that people's performance would be the same on each exam? Have they ever been compared directly or have many people taken both tests? If not, how can we possibly predict one from the other?

We should also be concerned about the degree of uncertainty associated with any point estimate. Traditionally prediction intervals are calculated to quantify this uncertainty. Is the margin of error sufficiently small (something that is context specific) that we would be comfortable using the point estimate?

You should also be concerned about the suitability of the process because the type of transformation you describe is accounting for sampling error which rests on the assumption of randomly selected samples from a population. I doubt that either exam satisfies that assumption, although they may approximate it.

| cite | improve this answer | | | | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.