Correspondence between two different depression scales?

Question

I have become involved in a project where psychiatrists are studying the correspondence between two scales measuring the level of depression among a specific patient group. One of the scales is for self-assessment and the other is used by the psychiatrists. The basic aim is to study the degree of correspondence between those scales. In a preliminary retrospective ve study, it turned out that there were gender differences with respect to the gap between patients' own perceptions and the expert opinions. This naturally made the researchers look for ways to take include such differences in the planned prospective study.

Since psychometrics is not my field, I don't see how such analyses are commonly done. And the psychiatrists involved don't seem to know any more than I do. So suggestions would be most welcome. My first thoughts have been to use a straightforward regression approach, regressing one scale on the other and gender and possibly other variables as independents. Comments? Suggestions for other approaches?

---

The original question was raised some time ago, but the problem has now surfaced in another setting, so I'll just continue here. Even though Sperman's rho or Kendall's tau would be possible to calculate, it still does not seem clear that these measures would answer the question at hand.

One reason is the criticism raised by e g Bland & Altman against using the correlation coefficient when comparing two measurements. One of several arguments is that you could get a high correlation while the agreement is poor, simply because there is high variability between subjects. So having larger samples which would result in greater variability would lead to a higher correlation.

Their setting is different, the question is rather the agreement between measurements of the same trait on the same scale, not different scales. But the argument against using correlation as a measure of agreement still seems valid, doesn't it?

Answer 1

Let's say you have a bivariate sample of scores $\{ (s_{11}, s_{12}), (s_{21}, s_{22}), \ldots (s_{n1}, s_{n2}) \}$. Your question can be rephrased as one about concordance, or the degree to which the two scores $\{ s_{1i} \}_{i=1}^{n}$ and $\{ s_{2i} \}_{i=1}^{n}$ "move together." In other words, do they rank patients in similar ways?

There are various ways you can measure this such as Spearman's $\rho$ or Kendall's $\tau$. Spearman's $\rho$ is an adaptation of Pearson's correlation which is based on the ranks of the observations instead of the observations themselves. Kendall's $\tau$ measures the probability that two variables are concordant (they both increase or both decrease across two independent samples) minus the probability that they're discordant. Each measure is contained in $[-1, 1]$ with perfect concordance occuring at $1$ and perfect discordance occuring at $-1$.

So returning to your problem, you might choose one of these statistics to determine the extent to which the self assessments agree with experts in terms of who is more or less depressed. I would suggest constructing a confidence interval based on one of these values and see if it contains zero. If it contains only values close to $1$ then you can conclude that there is strong correspondence between the scores.