I have one questionnaire with 12 questions. Each question has approximately 5 rating scales to choose from (categorical but not likert). There will be 20 raters who will each complete 3 questionnaires based on 3 different scenarios. What is the best measure of inter-rater reliability for this metholodogy, Kappa or ICC? And, what is the best way to structure the data for use with either SPSS or STATA (i.e. use numeric codes for the rating scales, put coders in rows or columns, etc.)? Thanks!
$\begingroup$
$\endgroup$
2
-
$\begingroup$ Your use of terms is unclear; please clarify and then I can likely answer. In the first sentence you state that you have one questionnaire, but in the third sentence you state that raters complete three questionnaires. Also, you state that each question has 5 rating scales that are categorical but not likert. Are you saying that each question has raters choose one of five nominal categories once, or that they do this five times for different sets of nominal categories? Finally, are you comparing ratings within or between scenarios? $\endgroup$– Jeffrey GirardSep 29, 2016 at 14:06
-
$\begingroup$ Thanks for replying, Jeffrey. I'll try to clarify. I have one questionnaire that consists of 12 questions with multiple choice responses. Only one response may be chosen for each question. I've assigned an ordinal numeric value to each response. There are 20+ raters that are reading 3 scenarios and completing one questionnaire for each scenario. My goal is to measure the inter-rater reliability of each question on the questionnaire. The hypothesis is that all raters should choose the exact same answer for each question. $\endgroup$– Catherine RSep 30, 2016 at 22:30
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
4
If I am understanding correctly, you have 20 raters that each complete a 12-item questionnaire on 3 different occasions; thus, each rater completes a total of 36 items. On each item, raters have to choose a single category from 5 possible categories; these categories exhibit an ordinal (i.e., ranked) relationship. Your goal is to determine the amount of agreement between the raters on each item, accounting for the fact that the categories have an ordinal relationship and that there are many raters.
This problem can be solved using a generalized agreement index (such as the generalized kappa coefficient, the generalized pi coefficient, or the generalized S score), the RWG index, or (depending on the level of observed between-rater variance) an intraclass correlation coefficient. You can read more about these options in the citations listed below; of them, I would recommend the RWG index if you think your categories are roughly interval (i.e., evenly spaced) or, if not, then the generalized S score with ordinal weights. You should prepare your data in a $36\times20$ matrix where each row corresponds to an item and each column corresponds to a rater. Within each cell (e.g., row $i$ and column $j$), put a numerical code corresponding to the category that rater $j$ assigned to item $i$.
You can calculate the generalized S score "by hand" using the formula provided here or you can click here to get functions to calculate it in SAS, R, MATLAB, or Excel (this one costs money). Note that the S score is also called the "Brennan Prediger" or "BP" index in some functions. I would also be willing to calculate the S score for you if you posted/sent me the matrix I described above.
A few concluding thoughts... You didn't explicitly mention wanting to compare reliability between the different measurement occasions (i.e., scenarios) but you could do so by calculating reliability for three tables each corresponding to those scenarios (e.g., rows 1-12, 13-24, and 25-36). You could also compute confidence intervals for each and then compare them. You also mentioned in your comment that you had a hypothesis that raters would choose the exact same answer. If you truly want them to choose the exact same answer, then your categories should be treated as nominal and not ordinal; with ordinal categories, raters get partial credit for getting close. Also, note that the goal of inter-rater reliability is not to test the statistical hypothesis that reliability is significantly different from zero. Rather, think of it as quantifying the extent to which they were reliable.
References
Gwet, K. L. (2014). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters (4th ed.). Gaithersburg, MD: Advanced Analytics.
James, L. R., Demaree, R. G., & Wolf, G. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69(1), 85–98.
McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30–46.
answered Oct 1, 2016 at 22:54
-
$\begingroup$ Thanks again, Jeffrey. You're advice is proving quite helpful. I've tried running the ICC in SPSS using the recommended 36 X 20 matrix. The result was a high level of reliability ICC=.979. However, I'd like to examine the reliability for each question separately. I've tried running the ICC for each question (Q1 through Q12) using a 3 X 20 matrix (i.e. Q1_scenarioA, Q1_scenarioB, Q1_scenarioC X 20 raters). This appeared to work well, but I want to make sure I have enough data to do this considering there are only 3 scenarios. Most of the examples I've seen have between 10 and 30 items in rows. $\endgroup$ Oct 4, 2016 at 23:24
-
$\begingroup$ You can do it as a descriptive and exploratory exercise, but it's probably not worth publishing on. If you calculate a confidence interval for these ICCs, you will notice that they are very wide with only 3 rows. So the estimate may help you identify problems and improve coding but you can't be very confident in it. $\endgroup$ Oct 5, 2016 at 2:56
-
$\begingroup$ Perfect. I think our overall ICC is supportive enough for what we're doing. I just want to ensure we haven't missed anything. Thanks again for your support. $\endgroup$ Oct 6, 2016 at 16:21
-
$\begingroup$ No problem. If you are happy with the answer, please mark it as "accepted" and consider upvoting it. This will help others find the answer. $\endgroup$ Oct 6, 2016 at 16:22