I am comparing two ordinal variables along a Likert scale. From previous help received on this forum, I have decided to use the Spearman Rho for my correlation statistic. My next question is how to test significance. I have used a Chi-squared, but there is also a significance test for the actual Spearman Rho statistic. I'm not quite clear on what the two should be telling me. I know the Chi-squared should tell me whether there is a significant relationship at all, whereas the Spearman Rho p-value will tell me if the correlation coefficient is significant, so please don't give me answers along that line. Is it possible to have one be significant but not the other?
closed as unclear what you're asking by SmallChess, jbowman, Nick Cox, Peter Flom♦ Jan 26 '18 at 13:49
Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.
No two different tests can reasonably be expected to yield the same probability of significance, that is implicit to their being genuinely different. The Pearson Chi-squared test implies the four assumptions of
(1) A simple random sample, that is, the sample data is a random sampling from a fixed distribution or population where every collection of members of the population of the given sample size has an equal probability of selection. Variants of the test have been developed for complex samples (2) A sufficiently large sample size. If a chi squared test is conducted on a sample with a smaller size, then the chi squared test will yield an inaccurate inference, e.g., committing a Type II error. (3) Sufficiently large expected cell counts. Some require 5 or more, and others require 10 or more. A common rule is 5 or more in all cells of a 2-by-2 table, and 5 or more in 80% of cells in larger tables, but no cells with zero expected count. When this assumption is not met, Yates's correction is applied. (4) Independence. The observations are always assumed to be independent of each other. This means chi-squared cannot be used to test correlated data (like matched pairs or panel data).
Since a certain frequency of data in one category correlates by subtraction to the other categories for the question at hand, condition (4) above, independence, is not met exactly leading in the binary response case to the Cochran's Q test. For an ordinal response, the Friedman test and the Durbin test are considerations.
By way of contrast, the Spearman's rho significance tests the strength of monotonicity of ranked data against no monotonicity and in so doing it adjusts for monotonic nonlinearity in the data. It seems to be the more appropriate test than Chi-squared. It is not only possible for one of these tests to be significant and the other not significant, but which is which is no doubt dependent on the characteristics of the data themselves, e.g., appropriateness of assumptions being a common explanation, another consideration the power of each test, but noise is another factor. That is, which is more significant can be, but does not have to be, due to chance alone.
Power for the Spearman's rho is almost identical to the Mann-Kendall test for detecting monotonicity in time series, also see corrigendum. The power of the Mann-Kendall test, in turn, is competitive with t-testing of slope, i.e., slight worse for linear conditions (as expected) and more powerful for nonlinearity. Power for the Friedman's test is modest and similar to Cochran Q is equivalent to the sign test's power. Thus, overall, I would expect Spearman's rho to be a better discriminator of significance than Friedman's test, in those circumstances where there is actual monotonicity.
I know the Chi-squared should tell me whether there is a significant relationship at all, whereas the Spearman Rho p-value will tell me if the correlation coefficient is significant, so please don't give me answers along that line. Is it possible to have one be significant but not the other?
If you really knew the first part, you would not need to ask the question.
The chi-square statistic ignores the ordinality of the data. The rank correlation does not. Either might be what you want, and it is certainly possible for one to be significant and the other not. There are also other statistics that use ordinal data; I like the Jonckheere-Terpstra test which seems to me to look at "ordinality" most strictly.