Skip to main content
3 of 3
deleted 5 characters in body
Carl
  • 13.3k
  • 7
  • 55
  • 115

No two different tests can reasonably be expected to yield the same probability of significance, that is implicit to their being genuinely different. The Pearson Chi-squared test implies the four assumptions of

(1) A simple random sample, that is, the sample data is a random sampling 
 from a fixed distribution or population where every collection of members 
 of the population of the given sample size has an equal probability of
 selection. Variants of the test have been developed for complex samples

(2) A sufficiently large sample size. If a chi squared test is conducted on
 a sample with a smaller size, then the chi squared test will yield an 
 inaccurate inference, e.g., committing a Type II error.

(3) Sufficiently large expected cell counts. Some require 5 or more, and
 others require 10 or more. A common rule is 5 or more in all cells of a
 2-by-2 table, and 5 or more in 80% of cells in larger tables, but no cells
 with zero expected count. When this assumption is not met, Yates's 
 correction is applied.

(4) Independence.  The observations are always assumed to be independent of 
 each other. This means chi-squared cannot be used to test correlated data
 (like matched pairs or panel data). 

Since a certain frequency of data in one category correlates by subtraction to the other categories for the question at hand, condition (4) above, independence, is not met exactly leading in the binary response case to the Cochran's Q test. For an ordinal response, the Friedman test and the Durbin test are considerations.

By way of contrast, the Spearman's rho significance tests the strength of monotonicity of ranked data against no monotonicity and in so doing it adjusts for monotonic nonlinearity in the data. It seems to be the more appropriate test than Chi-squared. It is not only possible for one of these tests to be significant and the other not significant, but which is which is no doubt dependent on the characteristics of the data themselves, e.g., appropriateness of assumptions being a common explanation, another consideration the power of each test, but noise is another factor. That is, which is more significant can be, but does not have to be, due to chance alone.

Power for the Spearman's rho is almost identical to the Mann-Kendall test for detecting monotonicity in time series, also see corrigendum. The power of the Mann-Kendall test, in turn, is competitive with t-testing of slope, i.e., slight worse for linear conditions (as expected) and more powerful for nonlinearity. Power for the Friedman's test is modest and similar to Cochran Q is equivalent to the sign test's power. Thus, overall, I would expect Spearman's rho to be a better discriminator of significance than Friedman's test, in those circumstances where there is actual monotonicity.

Carl
  • 13.3k
  • 7
  • 55
  • 115