Mann-Whitney U test and paired data

Question

I've read that working with paired data requires other method than Mann-Whitney $U$ test. I am not sure about how formally accurate is this statement.

Reasons:

Suppose that there is a set of $n$ (for a very large $n$) systems $S=\{s_1, s_2, ... , s_n\}$. In each system there is a property, and the exact value for the property in the system $s_i$ is a well defined and constant value $y_i$.

Suppose that there are two imperfect instrument $A$ and $B$ for the measurement of the property related to the $\{y_i\}$ values. The respective values of the measurements of $y_i$ are $x_{A,i}$ and $x_{B,i}$. And also suppose that the measurements errors of both instruments are independent of the $y_i$ value in the range of $y_i$ values found in $S$.

If I am right, in principle, we can take two randomly chosen subsets (of $m$ and $l$ elements, with $20 < m$, $l \ll n$) of $S$, $S_A$ and $S_B$, and analyze the distributions of the $\{x_{A,i}\}$ found in $S_A$ and the $\{x_{B, i}\}$ found in $S_B$ by comparing them with the Mann-Whitney $U$ test.

Depending of the chosen $S_A$ and $S_B$ we can get different results ($U$ and $p$-value). The key point here is: If we take $S_A = S_B$, we can pair data ($x_{A,i};x_{B,i}$) and use the Wilcoxon signed-rank test. If $m = l$, each randomly chosen $S_i$ has the same occurrence probability, and $S_A = S_B$ should be a valid choice.

If so, the Mann-Whitney $U$ test can be used for the comparison even if the data can be paired.

Questions:

• Is the reasoning above correct? If so, which is the advantage of using Wilcoxon signed-rank test instead of Mann-Whitney $U$ test? Is it related to confidence?
• If it is wrong, where is the mistake?
• What exactly does the $p$-value mean in the case of the Wilcoxon signed-rank test?

Answer 1

1. (I'm not sure I really follow your reasoning.) The Mann-Whitney U-test can be used with paired data. It will simply be less powerful. When you ignore the pairing, you are throwing a lot of information away.
2. I don't really understand this question.
3. The meaning of p-values here is the same as the meaning of p-values anywhere in frequentist statistics. That is, it is the probability of finding data as far or further from the null value if the null hypothesis is true. It may help you to read this CV thread: What is the meaning of p values and t values in statistical tests?

Answer 2

I mean if it violate some assumptions of the test

Certainly it violates assumptions, because any machine that crops up in both samples will have scores for the two measures that are dependent (due to the impact of the unobserved $s_i$), when there is an explicit assumption is of independence.

This will impact the behavior of the test.

If so, which is the advantage of using Wilcoxon signed-rank test instead of Mann-Whitney U test?

If dependence is substantial, the true significance level of the rank-sum test may be severely impacted (simulation in the case of paired data indicates the effect can be quite strong). By contrast, the signed rank test is designed for the situation, and it has better power in the presence of dependence (I'd guess largely due to the fact that its significance level isn't pushed down). If the dependence is low (e.g overlapping samples with very small overlap), it won't matter so much.

If you do have overlapping samples and you know which ones are paired, you could separate into "paired" and "independent" subsets, apply signed rank to the paired and rank-sum to the unpaired and combine the two p-values (say via Fisher's method).

If you have the option, take advantage of the pairing, and use all-paired data.

What exactly does the p-value mean in the case of the Wilcoxon signed-rank test?

The same as what it means for any other hypothesis test. See the second sentence of the second paragraph here

[the] p-value is the probability of obtaining the observed sample results, or a "more extreme" result, when assuming the null hypothesis is actually true