7
$\begingroup$

McManus (2012) discusses the three standard errors of measurement and provides the following definitions.

$SE_{meas} = SD \sqrt{(1 - reliability)}$
The standard error of measurement is an estimate of the validity of the actual scores likely to be obtained given a candidate’s true score (which is not and cannot be known).

$SE_{est} = SD \sqrt{(reliability × (1 - reliability))}$
The standard error of estimation is an estimate of the variability of a candidate’s true score, given their actual score.

$SE_{pred} = SD \sqrt{(1 - reliability^2)}$
The standard error of prediction is an estimate of the variability of a candidate’s actual score when taking a test a second time, given their performance on the first test.

One thing I find mystifying is SEpred reaches the point of maximum divergence from SEest at around .6 reliability. Here's image from a little Excel chart I made that illustrates this phenomenon.

enter image description here

I can follow the definitions but the article doesn't really explain why the calculations should be different in the different contexts. Is anyone able to provide an explanation?

McManus, I. C. (2012). The misinterpretation of the standard error of measurement in medical education: A primer on the problems, pitfalls and peculiarities of the three different standard errors of measurement. Medical Teacher, 34, 569-576.

$\endgroup$

3 Answers 3

4
+100
$\begingroup$

Okay, I'm not sure if anyone's still checking this (since it's from a year ago), but here's my response to your question: "I can follow the definitions but the article doesn't really explain why the calculations should be different in the different contexts. Is anyone able to provide an explanation?"

The calculations are different because of the purpose of each of these measurements. I'll go through them one by one. For each one, let's consider you taking a test. We know that in the general population, the average score for a certain measurement equals a specific value (let's say 100, to keep it simple). There's an associated standard deviation (let's say, 15, to keep it consistent with your chart) that describes how the data is distributed around that mean. So if you measure and graph a large population, you should get a curve centered at 100 with a certain spread defined by the standard deviation.

Let's also define a few terms, to make sure we're clear on what they all mean:

"True score" - the sample's true value for this score (i.e. what a perfectly accurate machine will spit out as a value when applied to this sample) "Actual score" - a machine's actual output when applied to a sample

Standard Error of Measurement

This is basically a measurement of how accurate the machine's scoring is. If, for example, a sample's true score was 90, a perfectly accurate machine would give you a score of 90 every time. The lower the reliability of the machine, though, the more varied would be the responses when you measured the sample. A somewhat accurate machine might give you scores of 85, 87, 91, 90, 92 for five tries. A less accurate machine might give you 93, 81, 96, 88, 89 for 5 tries. Consider this as a new curve based off of measuring the same person a bunch of times. That person has one "true score", but the machine will create a spread of "Actual scores". The less reliable the machine, the more spread out the actual scores.

Standard Error of Estimation

This description is a little confusing. It says "The standard error of estimation is an estimate of the variability of a candidate’s true score, given their actual score." A true score doesn't vary - it's fixed. I think what this is trying to say is if you have a bunch of people with the same actual score, then this measurement would be a measurement of the variability of their true scores. Here, the more reliable the machine, the less variation among those people's actual scores. This is also (though counterintuitively for many people) true if the measurement machine is really unreliable. If the machine is really unreliable, we basically have no idea where those people should really be scoring, so they are all likely coming from the middle of the true distribution (that's where the most people are) and so there won't be as much variation among them.

Standard Error of Prediction

The standard error of prediction can be pictured this way. You take one measurement of the sample with your machine. Let's say it gives you a value of 95. You then say, "I predict that if I measure this sample again, I will get a score of x." If the machine is perfectly reliable, you can say you'll get a score of 95. But the more unreliable the machine, the less likely your prediction. Most often, you'd say something like "I predict that if I measure this sample again, I will get a score between x and y." The less reliable the machine, the wider range you have to give to be confident in your prediction. It's higher because, as you say in a comment above, you have two sources of unreliability - your initial measurement and the second (upcoming) measurement.

I hope that his helps (and gets seen!).

$\endgroup$
2
$\begingroup$

Only the first of those is a standard error of measurement. Calling all three of them by the same name only confuses things. See sec 3.8 Errors of Measurement, Estimation, and Prediction, pp 66-69 in Lord & Novick's Statistical Theories of Mental Test Scores, Addison-Wesley, 1968.

$\endgroup$
1
  • $\begingroup$ While the terminology may be suboptimal it is common. The oft-cited reference Dudek (1979) talks about "our three formulae for the three standard errors of measurement". $\endgroup$ Jun 2, 2013 at 11:21
0
$\begingroup$

You are measuring 3 differents things, why do you expect to have only one standard error ? Intuitively, you will probably have differents means. This is the same thing for standard error.

With your exemple you can see that in the thrid process you take 2 measures, this is likely to be more reliable then taking only one measure.

The calculation is different because the context is different. From Wikipedia: "The standard error is the standard deviation of the sampling distribution of a statistic.[1] The term may also be used to refer to an estimate of that standard deviation, derived from a particular sample used to compute the estimate." If you estimate differents things, you will have differents standards errors. The expression of the standard error will depend on your measure.

$\endgroup$
2
  • $\begingroup$ "With your exemple you can see that in the thrid process you take 2 measures, this is likely to be more reliable then taking only one measure." The SEpred is always equal to or higher than the other two. My understanding is that this is because taking the test twice results in an additional source of measurement error. $\endgroup$ May 28, 2013 at 9:53
  • $\begingroup$ Your understanding is better than mine as it match your formulae. But for me this is counter intuitive. I would be glad to take a look at your source. $\endgroup$ May 28, 2013 at 10:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.