I have become involved in a project where psychiatrists are studying the correspondence between two scales measuring the level of depression among a specific patient group. One of the scales is for self-assessment and the other is used by the psychiatrists. The basic aim is to study the degree of correspondence between those scales. In a preliminary retrospective ve study, it turned out that there were gender differences with respect to the gap between patients' own perceptions and the expert opinions. This naturally made the researchers look for ways to take include such differences in the planned prospective study.
Since psychometrics is not my field, I don't see how such analyses are commonly done. And the psychiatrists involved don't seem to know any more than I do. So suggestions would be most welcome. My first thoughts have been to use a straightforward regression approach, regressing one scale on the other and gender and possibly other variables as independents. Comments? Suggestions for other approaches?
The original question was raised some time ago, but the problem has now surfaced in another setting, so I'll just continue here. Even though Sperman's rho or Kendall's tau would be possible to calculate, it still does not seem clear that these measures would answer the question at hand.
One reason is the criticism raised by e g Bland & Altman against using the correlation coefficient when comparing two measurements. One of several arguments is that you could get a high correlation while the agreement is poor, simply because there is high variability between subjects. So having larger samples which would result in greater variability would lead to a higher correlation.
Their setting is different, the question is rather the agreement between measurements of the same trait on the same scale, not different scales. But the argument against using correlation as a measure of agreement still seems valid, doesn't it?