Comparing 2 Likert Scales for 1 Population

Question

I have two 5-point Likert scales (Strongly Disagree to Strongly Agree) and I want to compare the results within one population sample. I expect participants to choose 'agree' or 'strongly agree' on both scales.

My sample size is small and in my study, I have assumed that my data is not normal so I have utilised non-parametric tests throughout.

I have read that the Wilcoxon Signed-Rank Test can be performed, however, I have also read that this test compares one population in two situations i.e. before and after an intervention. This is not what I am after.

I have read about correlation (such as Spearman's rank) but a significant number of assumptions must be met.

Any comments or suggestions are appreciated.

Answer 1

The most common approach to your query would be to apply a test or a confidence interval to the Spearman correlation coefficient. Given a sample of pairs $(y_i,x_i),\, i=1,\ldots,n$, of continuous variables, the Pearson correlation coefficient is

$$ r_p = \frac{\sum_i(x_i-\bar x)(y_i -\bar y)}{\sqrt{\left[\sum_i(x_i-\bar x)^2\right]\left[\sum_i(y_i-\bar y)^2\right]}}. $$

The Spearman correlation coefficient, say $r_s$, is obtained by replacing $(y_i,x_i)$ in $r_p$ by their respective ranks, or the mid ranks since here you have got ties.

You can accomplish this in R by the command

cor.test(rank(y), rank(x), conf.int = TRUE)

Alternatively, you may consider computing an agreement coefficient. Agreement coefficients are useful when you want, for instance, to assess if two instruments agree. For example, you may have two assays designed to measure the same thing, but neither is the gold standard. In this case, even if both assays were perfectly linearly related or perfectly correlated, they do not perfectly agree or yield identical answers, unless the line relating them is the identity line. For more details on this, check Lin (1998) A concordance correlation coefficient to evaluate reproducibility, Biometrics 45, 255-268 and Fay (2005) Random marginal agreement coefficients: rethinking the adjustment for chance when measuring agreement, Biostatistics 6, 171-180.