I am comparing two ordinal variables (i.e., two independent Likert items). I've used a chi-squared test to test significance of a relationship between the two, and a Pearson correlation (I'm expecting the two variables to have a linear relationship) to test the strength. I have a sample size of 250 and I'm looking at a table running 4x5 (one ordinal item has four integers, and the other has 5). Admittedly, not all cells in the contingency table have a value of at least 5. After running the tests, I got a very significant p-value for the chi-squared (~10^-7), but the correlation coefficient is quite low (~0.2). Can someone explain to me how this makes sense? Or am I using the wrong methods? What would be a better method to use?

  • 3
    $\begingroup$ As your sample size increases smaller and smaller effects become "statistically" significant. $\endgroup$ – mdewey Jul 6 '17 at 7:12
  • $\begingroup$ What is the scoring system you have used for Liberton items that are being referrred. ? What are you trying to ascertain ? Please be specific. $\endgroup$ – Subhash C. Davar Jul 8 '17 at 9:42
  • $\begingroup$ @subhash c. davar Both are just integer values. For instance, a rating of pain from 1 to 5. $\endgroup$ – Byakko Jul 10 '17 at 14:26
  • $\begingroup$ Apparently, you have termed incorrectly the two variables as ordinal variables. Would you like to present your contingency table ? How did you classify the data ? Pearson correlation doesn't test the strength ! Your claim is invalid. The computation of Chi - square statistic is not possible with the kind of data you have! $\endgroup$ – Subhash C. Davar Jul 10 '17 at 15:03
  • 1
    $\begingroup$ I'm not quite sure what you mean by "test" the strength. It is true that correlation reflects the strength, and that you can do a test of significance on the correlation coefficient itself. Ordinal refers to any categorical variable that has an order, such as the Likert scale. I think you may be confusing this with a continuous integer scale, but with Likert, there is no set way to measure the differences between 1 and 2, 2 and 3, and so on. $\endgroup$ – Byakko Jul 11 '17 at 1:54

Erring on the side of being pedagogical, Pearson correlation is not recommended for ordinal variables. Even after considering the Likert scales aspect of your data I would still be wary of using Pearson's because of the number of assumptions it requires.

Investigate Spearman or Kendall's correlations instead for effects like whether or not both measures are given by every person i.e. paired etc.

Another similar question that might be useful.

  • $\begingroup$ Hi, thank you for the response. I was under the assumption that Spearman was basically the same as Pearson, except that you convert the data to ranks prior to the Pearson analysis. Which in this case seems to be redundant as the ordinal variables are already discrete integer ranks?? Please let me know if I'm misunderstanding this. $\endgroup$ – Byakko Jul 6 '17 at 4:38
  • $\begingroup$ Spearman rank correlation does not assume similarity of variances. A quick scatterplot will give more clarity. In general, Pearson measures proportional increase/decrease in one measure with another. Spearman measures monotonic increase/decrease with some leeway for the shape of the data. Think of fitting a smoothing curve on the x,y scatterplot of these variables, if it's not a straight line, Spearman will be a better fit. Do not discount Kendall's. You can see the actual underlying code for these methods by just typing cor on the R console. $\endgroup$ – DivyaJyoti Rajdev Jul 6 '17 at 4:54
  • $\begingroup$ Thank you so much. I was using Excel before, but I'll switch to R. $\endgroup$ – Byakko Jul 6 '17 at 5:25
  • $\begingroup$ Sorry, I have two other questions. I've switched to using Spearman's, but the values are fairly similar to the Pearson's. My first question is: What is the difference between using a Chi-squared test vs the Spearman Rho's test? And my second question is: Why are the correlation coefficients so low when the Chi-squared test looks significant? $\endgroup$ – Byakko Jul 6 '17 at 14:02

My first question is: What is the difference between using a Chi-squared test vs the Spearman Rho's test?

The chi-squared test treats both variables $X$ and $Y$ as nominal (like colors, countries etc.) and thus, it can detect any sort of underlying relationship between $X$ and $Y$. In contrast, tests for linear resp. rank-correlations make use of the fact that the factor levels are ordered and are particularly suitable to detect linear resp. monotone underlying relationships. In your setting, it seems to make sense to condense the relationship to a correlation coefficient, so it would be more natural to provide p values associated with that measure instead of the less focused chi-squared test. But basically it is up to you.

Of course you can't just run all tests that you know and then pick the one with the smallest p value. Ideally, you already select an analysis strategy before looking at the data to avoid data snooping and to end up with reproducible conclusions.

PS: You are also free to use linear correlations instead of rank-correlations. Its test is basically the "linear-by-linear" test for association by Agresti, one of the godfathers of modern categorical data analysis. If you are interested, his famous book [1] is worth every penny. You will find it in every university library.

And my second question is: Why are the correlation coefficients so low when the Chi-squared test looks significant?

A small p value means strong evidence against the null hypothesis "no relationship between $X$ and $Y$". Depending on the sample size, a sample correlation of 0.2 can mean extremely strong evidence or, if the sample is small, not much evidence against this null hypothesis. Or in other words: the p value is not a measure of effect size.

[1] Agresti, A. (2002). Categorical Data Analysis, Second Edition. Hoboken, New Jersey: John Wiley & Sons.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.