How to measure correlations/congruence between parent and teacher ratings of children's personality

Question

I was wondering if anyone could help. I am investigating children's personality, and I have collected both parents' and teachers' ratings for children on a 24-item personality scale of the big-five (conscientiousness, extraversion, openness, agreeabless and emotional-stability). I wondered if anyone knew the best technique to examine the correlation/congruence between the parents and teachers ratings across children. Ideally, I'd like to do this for each of the 24-items individually, but also for each of the five traits themselves (each trait has 4-6 questions, so for example, 4 questions make up extraversion, 5 agreeableness, etc.). My goal is to have a picture of how well teachers and parents agreed on each item, as well as on each of the five traits themselves.

Any help would be greatly appreciated.

Best,

Bruce

Answer 1

Note that the question is phrased as visualization, not as a falsifiable hypothesis. To visualise, you can plot % agreement (for binary) or level of disagreement (difference between the two scores for each child, for ordinal) for each sub/category with appropriate measure of dispersion, block by color for trait, that's it.

If, however, you would like to test for significance of differences, you can explore a range of solutions depending on your more specific question, levels of responses and the range of complexity you want to accommodate: e.g. inter-rater reliability approaches if the focus is on the agreement; using Likert scale approaches, expressing agreement rates as percentages for each trait and comparing them using a regular ANOVA (if focusing on which trait has significantly higher rates of agreement). Directionality may be of interest, too, it seems, to see whether one group tended to give higher scores.

Answer 2

For each item and for each trait, create an $n$-by-$r$ matrix where $n$ is the number of children and $r$ is the number of raters (e.g., $r=2$ if there are ratings from one parent and one teacher per child). Each row should correspond to a child and each column should correspond to a rater. Fill these matrices with the item and trait scores for each child from each rater. If the scores are continuous, you can use single-score, two-way intraclass correlation coefficients to quantify the consistency or agreement of the ratings. If the scores are categorical, you can use a weighted chance-adjusted agreement index such as Cohen's kappa, Fleiss' kappa, or Bennett et al.'s S score.