4
$\begingroup$

I've read that working with paired data requires other method than Mann-Whitney $U$ test. I am not sure about how formally accurate is this statement.

Reasons:

Suppose that there is a set of $n$ (for a very large $n$) systems $S=\{s_1, s_2, ... , s_n\}$. In each system there is a property, and the exact value for the property in the system $s_i$ is a well defined and constant value $y_i$.

Suppose that there are two imperfect instrument $A$ and $B$ for the measurement of the property related to the $\{y_i\}$ values. The respective values of the measurements of $y_i$ are $x_{A,i}$ and $x_{B,i}$. And also suppose that the measurements errors of both instruments are independent of the $y_i$ value in the range of $y_i$ values found in $S$.

If I am right, in principle, we can take two randomly chosen subsets (of $m$ and $l$ elements, with $20 < m$, $l \ll n$) of $S$, $S_A$ and $S_B$, and analyze the distributions of the $\{x_{A,i}\}$ found in $S_A$ and the $\{x_{B, i}\}$ found in $S_B$ by comparing them with the Mann-Whitney $U$ test.

Depending of the chosen $S_A$ and $S_B$ we can get different results ($U$ and $p$-value). The key point here is: If we take $S_A = S_B$, we can pair data ($x_{A,i};x_{B,i}$) and use the Wilcoxon signed-rank test. If $m = l$, each randomly chosen $S_i$ has the same occurrence probability, and $S_A = S_B$ should be a valid choice.

If so, the Mann-Whitney $U$ test can be used for the comparison even if the data can be paired.

Questions:

  • Is the reasoning above correct? If so, which is the advantage of using Wilcoxon signed-rank test instead of Mann-Whitney $U$ test? Is it related to confidence?
  • If it is wrong, where is the mistake?
  • What exactly does the $p$-value mean in the case of the Wilcoxon signed-rank test?
$\endgroup$
  • $\begingroup$ Why would you want to test these two data sets if the values are constant? $\endgroup$ – gung - Reinstate Monica Feb 19 '15 at 22:24
  • $\begingroup$ Because my interest is in measurement devices. The {y_i} can be unknown in other situations. The statement about constants is to make clear that the measurement do not disturb the systems. $\endgroup$ – user1420303 Feb 19 '15 at 22:44
  • $\begingroup$ Are you wondering how to do instrument calibration with non-normal data? $\endgroup$ – gung - Reinstate Monica Feb 19 '15 at 22:53
  • $\begingroup$ No, I wish yo be sure about the correctness of using Mann-Whiney test in data that can be paired. $\endgroup$ – user1420303 Feb 19 '15 at 23:49
  • $\begingroup$ What do you mean by "correctness" of the test in that comment? Do you mean "the test statistic has the usual distribution under the null", or something else? $\endgroup$ – Glen_b Feb 19 '15 at 23:59
3
$\begingroup$
  1. (I'm not sure I really follow your reasoning.) The Mann-Whitney U-test can be used with paired data. It will simply be less powerful. When you ignore the pairing, you are throwing a lot of information away.
  2. I don't really understand this question.
  3. The meaning of p-values here is the same as the meaning of p-values anywhere in frequentist statistics. That is, it is the probability of finding data as far or further from the null value if the null hypothesis is true. It may help you to read this CV thread: What is the meaning of p values and t values in statistical tests?
$\endgroup$
3
$\begingroup$

I mean if it violate some assumptions of the test

Certainly it violates assumptions, because any machine that crops up in both samples will have scores for the two measures that are dependent (due to the impact of the unobserved $s_i$), when there is an explicit assumption is of independence.

This will impact the behavior of the test.

If so, which is the advantage of using Wilcoxon signed-rank test instead of Mann-Whitney U test?

If dependence is substantial, the true significance level of the rank-sum test may be severely impacted (simulation in the case of paired data indicates the effect can be quite strong). By contrast, the signed rank test is designed for the situation, and it has better power in the presence of dependence (I'd guess largely due to the fact that its significance level isn't pushed down). If the dependence is low (e.g overlapping samples with very small overlap), it won't matter so much.

If you do have overlapping samples and you know which ones are paired, you could separate into "paired" and "independent" subsets, apply signed rank to the paired and rank-sum to the unpaired and combine the two p-values (say via Fisher's method).

If you have the option, take advantage of the pairing, and use all-paired data.

What exactly does the p-value mean in the case of the Wilcoxon signed-rank test?

The same as what it means for any other hypothesis test. See the second sentence of the second paragraph here

[the] p-value is the probability of obtaining the observed sample results, or a "more extreme" result, when assuming the null hypothesis is actually true

$\endgroup$
  • $\begingroup$ The phrasing for the meaning of the p-value from Wikipedia is rather awkward. I just tweaked it. $\endgroup$ – gung - Reinstate Monica Feb 20 '15 at 15:02
  • $\begingroup$ @gung it used to be much better, but it's one of those articles people can't leave alone whether they know what they're doing or not; I appreciate the effort. $\endgroup$ – Glen_b Feb 20 '15 at 15:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.