In my study a dog's owner and dog's walker each filled out a personality assessment for a target dog. I have 60 dogs in total. There were multiple questions that scored the dog on 5 dimensions. I'm comparing the inter-rater reliability of these assessments. I'm trying to determine if I should be using Pearson correlation or Intraclass Correlation (ICC). If I use ICC, (which I think I should be using) I'm trying to determine the following settings in R: one way or two way; agreement or consistency; average or single.
Rater reliability studies typically have a number of subjects (dogs) rated by a small number of raters. What you have looks like a multivariate repeated measures design. To keep things simple, you could look at each score on its own -- say extroversion -- and do a paired t-test on the scores (owner - walker for each dog).
I would argue that you want a two-way model because there is meaningful variance associated with both dogs and raters (a one-way model would only look at variance associated with dogs). Whether you want [agreement or consistency] and [single or average] depends on how you want to use the ratings. If you plan to use the ratings from a single rater for each dog, then calculate single score ICCs; if, however, you plan to use the average of multiple raters' ratings for each dog, then calculate average score ICCs. Finally, if you plan to use ranks or z-scores for analysis (and therefore don't care if raters have different means), then calculate consistency ICCs; if, however, you plan to use the raw scores (and therefore want raters to have the same mean), then calculate agreement ICCs.
McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30–46.