How does one check for interrater reliability when one is asked to run a qualitative analyses where each statement may tap into various themes? e.g.,
why do you like gardening:
Coder 1:
| Participant's response | theme 1 - exercise | theme 2 - relaxing | theme 3 - monetary theme | theme 4 - aesthetics |
|---|---|---|---|---|
| I like to garden because my house looks nicer and can sell some of the flowers | 0 | 0 | 1 | 1 |
| It's a nice form of exercise | 1 | 0 | 0 | 0 |
| Flowers are pretty | 0 | 0 | 0 | 1 |
Coder 2:
| Participant's response | theme 1 - exercise | theme 2 - relaxing | theme 3 - monetary theme | theme 4 - aesthetics |
|---|---|---|---|---|
| I like to garden because my house looks nicer and can sell some of the flowers | 0 | 0 | 1 | 0 |
| It's a nice form of exercise | 1 | 0 | 0 | 0 |
| Flowers are pretty | 0 | 0 | 0 | 1 |
How can we calculate the overall agreement across all themes and all participants between these coders? I know this is a silly example, I just wanted to be able to demonstrate what I mean. I use R for data analysis.
