4
$\begingroup$

Situation: A three part ability test with multiple-choice, essay, and oral components. I have scores for each of 110 test takers on each part and overall. The three parts are unequally weighted in computing the overall score. There is reason to think that the three parts measure different abilities and are not highly correlated.

Available information: internal-consistency reliability of the M/C test (.82), and interrater-reliability of the grading of the essay and oral components (both near .95). The two parts were graded by different graders.

Desired goal: Estimate test-retest reliability of overall score

Would the low reliability of .82 drive the reliability of the total score, or would the reliability of the total score be closer to a weighted average of the three reliabilities? Is there a formula for combining these reliabilities to get an estimate of the test reliability of the overall score? What else would I need to know or what assumptions would I need to make to use the formula? If this is a complex matter, can you refer me to a relevant textbook or journal article?

I found these two related questions on this site but the answers were not helpful to me: Reliability of Composite Variable Made of 4 Measures? How to assess the reliability of a composite scores?

I guess I could do a Monte Carlo study. My gut feeling is that the low reliability will drive the overall reliability. But is there an analytic approach that would give a more definitive answer?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
+100
$\begingroup$

Please find below the link to the Chapter on reliability that discusses many of your issues. For the specific issue of test-retest reliability see p.27-28, and reliability of a composite test is covered on page 32. This Chapter is written by William Revelle, a reputable psychometrician, hence the source is credible.

https://www.personality-project.org/r/book/Chapter7.pdf

Improve this answer
$\endgroup$
12
  • $\begingroup$ Are you referring to formula 7.38? (Nice chapter!) $\endgroup$
    – Joel W.
    Commented Oct 22, 2019 at 23:06
  • $\begingroup$ How do you understand the formula to work if the subtests are independent and standardized before combination? (Clearly the reliability of the sum is less than the reliability of the least reliable subjest.) What is the effect of the subtest and whole test variances in the situation I presented? $\endgroup$
    – Joel W.
    Commented Oct 22, 2019 at 23:34
  • $\begingroup$ @JoelW. yes that's right, equation 7.38 is composite reliability. To calculate it you would simply need reliability of each subtest (e.g. alpha), variance of each subset, and the total test variance. Then simply plug in the values in the formula and here you have composite reliability of your test. If you found this answer useful, I would be appreciative if you could accept it please $\endgroup$
    – PsychometStats
    Commented Oct 22, 2019 at 23:36
  • $\begingroup$ @JoelW. ok, there are two ways in which you can proceed, I will write them down for you in a moment $\endgroup$
    – PsychometStats
    Commented Oct 22, 2019 at 23:41
  • $\begingroup$ @JoelW. 1. I assume that M/C test is one of three subtests. Then, you can use reliability of each component to get total reliability. That is, internal consistency of the M/C (.82), inter-rater reliability for grading (about .95), and for oral (about .95). You also have the remaining parts, including variance of each subtest and total variance. This method is based in my opinion on a less stringent assumption of combining different types of reliability into composite reliability. I would not worry about standardisation, because it is common practice in reliability assessment. $\endgroup$
    – PsychometStats
    Commented Oct 22, 2019 at 23:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.