Help calculating Inter-rater reliability in a complex online survey (free text). The questions are free text. We have now coded the potential answers. Examples: 1.) Hobbies (Time in nature, mind-body practices, arts) 2.) Expectations about 5-MeO-DMT ceremony (Psychedelic experience, feeling oneness with others, relief of depression etc.,).
Two tricky things: 1.) There are a lot of questions (>50). Are we supposed to calculate the inter-reliability for each question? Or can there be a general inter-reliability calculation for the survey.
2.) Participants may be assigned to multiple answers for one questions i.e Participant #1 hobbies: time in nature , reading. Participant #2: time in nature, fashion, arts.
So not sure how the coding, inter-reliability calculation would work here
Thanks for any help!!