Independence test informs paired difference test in matched pre/post settings?

Question

I have a, perhaps faulty, intuition that if a paired difference test (e.g., Wilcoxon rank sum, t-test) for a pre/post item looks significant, one should use caution in interpreting this fact if an independence test (e.g., Kendall's tau-b, Spearman's rho, Pearson rank) looks insignificant. It's not that independence (association or correlation) has anything to do with differences, but that alternative explanations for the paired difference test (apart from treatment effects) seem to increase in likelihood relative to the conclusion, "Use our medicine, data suggests that the typical patient will do better in random (or unpredictable) ways."

While "caution" may be wrong, the question is: In what ways, if any, should independence tests and paired difference tests inform each other under the 4 possible significant/insignificant combinations on pre/post studies of treatment effects?

As concrete problem details may be useful (or bring up other problems), my practical problem includes the following:

• 10 - 40 observation pairs
• Items are "ordered 1 - 5," Strongly disagree to Strongly agree
• Limited knowledge of false negatives profiles for independence tests referenced

Answer 1

I think your intuition is on to something.

In general, you can't expect too much out of a simple statistical test. Basically what the signed rank test is telling you is that the change from Pre to Post was "on average" larger in one direction than the other.

In any real context, there is a lot more you want to know.

Plots will be helpful here.

As you suggest, it may make sense to plot Post vs. Pre. But I think it may make more sense to plot Change vs. Pre. This would more directly indicate what Change is expected for what level of Pre, which is what I think you may be getting at.

One useful plot is Post vs. Pre with a one-to-one line drawn in. Points above or to the left of the line indicate observations where Post is greater than Pre.

You can also plot a histogram-like bar plot of differences, which really summarizes the data that the test is evaluating.

This plot also allow you to tease out simple, important, and easily understandable statistics: What percentage of observations showed an increase? A decrease? Stayed the same? What percentage of observations showed an increase of 3 units? And so on.

In any real-world context, ideally there are other variables that could be looked at to help explain the observed effect. Combining these into a more complex model --- maybe mixed effects ordinal regression in your case --- may ultimately have better explanatory power than a simple test.