0
$\begingroup$

In my study a dog's owner and dog's walker each filled out a personality assessment for a target dog. I have 60 dogs in total. There were multiple questions that scored the dog on 5 dimensions. I'm comparing the inter-rater reliability of these assessments. I'm trying to determine if I should be using Pearson correlation or Intraclass Correlation (ICC). If I use ICC, (which I think I should be using) I'm trying to determine the following settings in R: one way or two way; agreement or consistency; average or single.

$\endgroup$
  • $\begingroup$ Do you have 60 owner's and walkers? One for each dog? $\endgroup$ – Placidia Aug 21 '15 at 18:20
  • $\begingroup$ Yes, each target dog has an owner and a walker. $\endgroup$ – user66492 Aug 21 '15 at 18:23
  • $\begingroup$ How are you going from the "multiple questions" to the "5 dimensions"? Is there an existing instrument for this? Are you doing a factor analysis (separately or combined)? Do you want to check agreement independently for each dimension? $\endgroup$ – gung Aug 21 '15 at 18:26
  • $\begingroup$ I'm using an existing dog personality instrument. I've calculated unit-weighted scores for each dimension. Yes, I'm looking at agreement for each dimension e.g., did both walkers and owners score the dog similarily on the extraversion dimension. $\endgroup$ – user66492 Aug 21 '15 at 18:32
1
$\begingroup$

Rater reliability studies typically have a number of subjects (dogs) rated by a small number of raters. What you have looks like a multivariate repeated measures design. To keep things simple, you could look at each score on its own -- say extroversion -- and do a paired t-test on the scores (owner - walker for each dog).

$\endgroup$
1
$\begingroup$

I would argue that you want a two-way model because there is meaningful variance associated with both dogs and raters (a one-way model would only look at variance associated with dogs). Whether you want [agreement or consistency] and [single or average] depends on how you want to use the ratings. If you plan to use the ratings from a single rater for each dog, then calculate single score ICCs; if, however, you plan to use the average of multiple raters' ratings for each dog, then calculate average score ICCs. Finally, if you plan to use ranks or z-scores for analysis (and therefore don't care if raters have different means), then calculate consistency ICCs; if, however, you plan to use the raw scores (and therefore want raters to have the same mean), then calculate agreement ICCs.

References

McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30–46.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.