In my research, I use a questionnaire so that students mention some classmates according to a group of characteristics. Each item is a different characteristic, and for each one they must name the classmate they consider to have it. In this case can I use Krippendorff's alpha to assess the reliability of the students' judgments?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
Krippendorff's alpha is a measure of (chance corrected) agreement. Reliability in classical test theory is the relationship between the true score and the measured score. However, measures such as Kripp alpha are often referred to as reliability (but maybe I'm just old).
However, you have a weird issue, that people (presumably) cannot nominate themselves, so 100% agreement isn't possible.
answered Jul 26, 2022 at 4:24
-
$\begingroup$ This means that in the first place one could not speak of reliability, since I am not trying to measure any construct, but rather the nomination of people who have a group of characteristics is sought. In this way, you could obtain the % of agreement between the participants, however this could not be 100% because they could not self-nominate. Is this the idea? $\endgroup$ Jul 26, 2022 at 15:02
-
$\begingroup$ Yeah, alpha wouldn't be horrible, just be aware it's going to be a little lower than it could be. $\endgroup$ Jul 26, 2022 at 16:09