3
$\begingroup$

Dudek (1979) describes the following formula as "the standard deviation of true scores when the observed score is held constant" (p. 337):

$SE_{est} = SD \sqrt{(reliability × (1 - reliability))}$

As I understand it, my true score is a hypothesized 'real' score for me on some test, for example an IQ test. Due to measurement error it won't necessarily be my actual score on any given test, but if I could repeat the IQ test infinitely many times with no practice effects then my true score would be the mode mean.

If I just have a single underlying true score, how can we talk about there being a standard deviation of true scores?

Dudek FJ. 1979. The continuing misinterpretation of the standard error of measurement. Psychol Bull 86:335–337.

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ Why the mode? With so little context it's hard to tell, but the standard deviation referred to sounds like perhaps it's across individuals at a given observed score. $\endgroup$
    – Glen_b
    Commented May 28, 2013 at 13:28
  • 1
    $\begingroup$ Since error is randomly distributed around 0, I think that my true score is the most likely value for me to obtain on any given repetition of the test. Hence I believe it's correct for me to equate the mode with the true score under the condition of infinite repetition that I described. I think it's plausible that the standard error of estimation relates to the spread of true scores across individuals who happen to have the same observed score. However, I couldn't find anything in the original Dudek paper or anywhere else to indicate that this was definitely true. $\endgroup$
    – user1205901 - Слава Україні
    Commented May 28, 2013 at 13:35
  • $\begingroup$ Using the mode might work but is not guaranteed to do so. For instance, the nature of the variation in your test scores might be such that sometimes you do slightly better than your true score but occasionally you do much worse. The mode will be an asymptotically biased estimator of your true score. In fact, because test results are necessarily discrete, then unless your true score coincides with one of the possible test values, the mode--whatever it might turn out to be in the long run--is guaranteed to be incorrect. $\endgroup$
    – whuber ♦
    Commented May 28, 2013 at 15:06
  • $\begingroup$ It seems that at minimum it would have been less confusing for me to instead equate the true score with the mean of the repeated scores. So I've edited the original post. $\endgroup$
    – user1205901 - Слава Україні
    Commented May 28, 2013 at 21:45

5 Answers 5

Reset to default
4
$\begingroup$

From a frequentist perspective, the parameter is fixed. Thus, from a frequentist perspective a person's true score is fixed (excluding of course for the moment that true scores can change over time). A standard error is the standard deviation of an estimator. Thus, the score you obtain for a person is an estimator of the true score. Typically in actual data analytic contexts, we don't have the standard error; instead, we have an estimate of the standard error. Thus, the estimated standard error of the true score is the estimated standard deviation of the estimator of the true score.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ fwiw this doesn't substantially differ from a Bayesian perspective: the person's true score is still fixed and a random (posterior) distribution describes your uncertainty about that fixed value, which has a mean and some std dev. On the other hand in actual data analysis contexts the posterior and thus its mean and std dev. are exact not estimated (modulo mcmc issues etc.) Upshot is that the Dudek quote just seems a bit misleading, though I suppose it might make more sense in context. $\endgroup$
    – conjugateprior
    Commented Jul 2, 2013 at 9:10
2
$\begingroup$

I've found an answer, although not one that I find particularly satisfying.

Here on p9 of the book "Handbook of Psychological and Educational Assessment of Children it is written:

"The standard error of estimation is used to establish a 'confidence interval', which indicates the probability that an individual's true score lies within a specified range over repeated 'theoretical' test occasions."

I still find this confusing. I'm used to building confidence intervals around unknown population means and stating that the confidence interval should be interpreted as a long-run frequency. My understanding is that it's incorrect to say "there's a 95% chance the mean is between X and Y" because the population mean is either the bounds of the confidence interval or it isn't. Aren't we just making the same mistake here?

Improve this answer
$\endgroup$
1
$\begingroup$

From what I can see, the equation $SE_{est} = SD\sqrt{reliability(1-reliability)}$ is an attempt to construct a measure more akin to a credibility interval used in Bayesian statistics, i.e. the probability of the parameter given the data. A confidence interval is, as you say, the long-run frequency - if you repeated the test a large number of times, the confidence interval would include the true score in 95% of those tests.

From the abstract of the Dudek article, it seems that people had used the formula $SE_{est} = SD\sqrt{1-reliability}$ to assess the variability in scores. Dudek argues that this gives the variability of the observed scores when the true score is held constant (which would be what we traditionally use for a confidence interval), but that what is actually desired is a measure of the variability of the true score when the observed score is held constant (at the measured value). Dudek states that "the [resulting] interval is around the estimated true score rather than around the observed score".

Reference: http://psycnet.apa.org/journals/bul/86/2/335/

Improve this answer
$\endgroup$
1
$\begingroup$

Good question. There is only true score for you, but there are many true scores for different test-takers. Dudek was probably referring to the distribution of true scores for all test-takers with the same observed score. See Richard A. Charter & Leonard S. Feldt, 2001, Confidence Intervals for True Scores: Is There a Correct Approach?, J. Psychoeducational Assessment 19:350–61.

Improve this answer
$\endgroup$
1
$\begingroup$

The standard error Dudek talks about is not for true scores but for estimated true scores. Estimated true scores are obtained by shrinking observed test scores towards the sample mean by an amount indicated by the error variance of the observed scores. The error variance of the estimated true scores is smaller than that of the observed scores (it's a case of bias-variance tradeoff). Lord & Novick's Statistical Theories of Mental Test Scores (1967) contains a good treatment of this topic.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.