3
$\begingroup$

Situation: A three part ability test with multiple-choice, essay, and oral components. I have scores for each of 110 test takers on each part and overall. The three parts are unequally weighted in computing the overall score. There is reason to think that the three parts measure different abilities and are not highly correlated.

Available information: internal-consistency reliability of the M/C test (.82), and interrater-reliability of the grading of the essay and oral components (both near .95). The two parts were graded by different graders.

Desired goal: Estimate test-retest reliability of overall score

Would the low reliability of .82 drive the reliability of the total score, or would the reliability of the total score be closer to a weighted average of the three reliabilities? Is there a formula for combining these reliabilities to get an estimate of the test reliability of the overall score? What else would I need to know or what assumptions would I need to make to use the formula? If this is a complex matter, can you refer me to a relevant textbook or journal article?

I found these two related questions on this site but the answers were not helpful to me: Reliability of Composite Variable Made of 4 Measures? How to assess the reliability of a composite scores?

I guess I could do a Monte Carlo study. My gut feeling is that the low reliability will drive the overall reliability. But is there an analytic approach that would give a more definitive answer?

$\endgroup$
1
+100
$\begingroup$

Please find below the link to the Chapter on reliability that discusses many of your issues. For the specific issue of test-retest reliability see p.27-28, and reliability of a composite test is covered on page 32. This Chapter is written by William Revelle, a reputable psychometrician, hence the source is credible.

https://www.personality-project.org/r/book/Chapter7.pdf

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Are you referring to formula 7.38? (Nice chapter!) $\endgroup$ – Joel W. Oct 22 '19 at 23:06
  • $\begingroup$ How do you understand the formula to work if the subtests are independent and standardized before combination? (Clearly the reliability of the sum is less than the reliability of the least reliable subjest.) What is the effect of the subtest and whole test variances in the situation I presented? $\endgroup$ – Joel W. Oct 22 '19 at 23:34
  • $\begingroup$ @JoelW. yes that's right, equation 7.38 is composite reliability. To calculate it you would simply need reliability of each subtest (e.g. alpha), variance of each subset, and the total test variance. Then simply plug in the values in the formula and here you have composite reliability of your test. If you found this answer useful, I would be appreciative if you could accept it please $\endgroup$ – PsychometStats Oct 22 '19 at 23:36
  • $\begingroup$ @JoelW. ok, there are two ways in which you can proceed, I will write them down for you in a moment $\endgroup$ – PsychometStats Oct 22 '19 at 23:41
  • $\begingroup$ @JoelW. 1. I assume that M/C test is one of three subtests. Then, you can use reliability of each component to get total reliability. That is, internal consistency of the M/C (.82), inter-rater reliability for grading (about .95), and for oral (about .95). You also have the remaining parts, including variance of each subtest and total variance. This method is based in my opinion on a less stringent assumption of combining different types of reliability into composite reliability. I would not worry about standardisation, because it is common practice in reliability assessment. $\endgroup$ – PsychometStats Oct 22 '19 at 23:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.