1
$\begingroup$

Help calculating Inter-rater reliability in a complex online survey (free text). The questions are free text. We have now coded the potential answers. Examples: 1.) Hobbies (Time in nature, mind-body practices, arts) 2.) Expectations about 5-MeO-DMT ceremony (Psychedelic experience, feeling oneness with others, relief of depression etc.,).

Two tricky things: 1.) There are a lot of questions (>50). Are we supposed to calculate the inter-reliability for each question? Or can there be a general inter-reliability calculation for the survey.

2.) Participants may be assigned to multiple answers for one questions i.e Participant #1 hobbies: time in nature , reading. Participant #2: time in nature, fashion, arts.

So not sure how the coding, inter-reliability calculation would work here

Thanks for any help!!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

  1. You're supposed to do the analysis that answers the questions that you have. If you want to know if there's rater agreement for each question, analyze each question. (You could also do something like take the average, but that could disguise some very bad questions. Another issue is that with 50 questions, you'd expect a lot of variation by chance.

  2. I would treat each one as a separate question. For Q1: Did raters agree about time in nature? Did raters agree about hobbies. Etc.

(The lesson to be learned here is to think about how you'll analyze the data before you collect it.)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.