0
$\begingroup$

I have 2 sets of paired values of time: A and B. I know that B is significantly higher than A using a Wilcoxon test (I have a small population) but I want to know which samples have a significantly different B. Using the difference B-A, I have the distribution of time change from which I have calculated a confidence interval (through boostrap method). I'm not sure about what threshold I should use to determine which samples actually have a significant change between A and B.

Thank in advance :)

$\endgroup$
2
  • $\begingroup$ I'm afraid that at least in classical frequentist statistics this question probably doesn't make sense. I don't think you can test individual pairs for significant differences. In a Bayesian framework you could probably determine the posterior distribution of the difference in each pair and see which differences have credible intervals that exclude zero ... $\endgroup$
    – Ben Bolker
    Commented Aug 4 at 18:11
  • $\begingroup$ Your question is unclear to me, perhaps because your terminology is non-standard. The "sets of paired values" would typically be called samples. What is the pairing? Is it pairing between observations in set A and set B or something else? When you ask for a way to identify "samples" that significantly differ, are you asking about individual data points, or some sort of sub-samples within the samples that you call "sets"? When you say "significantly" are you talking os statistical significance or something more substantive in the context of your study? $\endgroup$ Commented Aug 4 at 21:04

1 Answer 1

0
$\begingroup$

As best I can tell from your post, you have 2 paired samples, where the values are some measure of time.

Now, you mention running a Wilcoxon test. That is quite ambiguous. It could be a Wilcoxon signed rank test (WSRt), or a Wilcoxon rank sum test (aka Mann-Whitney U test MWUt). As you mention that your data is paired, I would assume WSRT, as the MWUt is for independent samples. The general consensus is that MWUt can not be used for paired samples, but there are differing opinions (e.g. see here on CV). So let's assume you ran a WSRt.

You say "I have a small population". Do you really mean "population", or did you mean "sample" (i.e. you have a small number of values in A & B)? Having a small sample size is not a very valid reason to use WSRt or MWUt, instead of a paired t-test; unless the differences ($B_i-A_i$)'s are very "non-normal" (which, with a small sample size, is pretty hard to prove or disprove), you will have more power with a paired t-test (which compares the mean of the differences to 0, or some other target).

You say that it showed "that B is significantly higher than A". But that is not what a significant WSRt result means. Generally, for a paired test, it says that the pseudomedian (aka Hodges–Lehmann estimator) of the differences ($B_i-A_i$) is not equal to $0$ (or some other target value). If the distribution of the ($B_i-A_i$)'s is symnmetrical, and only if it is symmetrical, does it then tell you if the median of the differences is equal to the target. But eyeballing, or even more testing for symmetry, with a small sample size is a bit of a futile exercise.

Now, the MWUt does tell you if one sample is "stochastically greater" than the other, which you imnply by saying "B is significantly higher than A". So this adds to the confusion as to which Wilcoxon test you used. Note that if you used the WSRt, it does not tell you anything about which sample is "greater" than the other, and if you used the MWUt, it is debatable that this is a valid test for paired data.

You say that you used a bootstrap method to obtain a CI for the differences. But if your sample size is small, bootstrap is suspect (the sampe is most likely too sample to be really representative).

Last, you seem to be asking "can I determine if a single difference ($B_k-A_k$) is somehow "significant". As a previous comment discussed, this does not make sense. This is trying to use statistics on a sample of size 1; pointless. Now, what you can do, assuming that the bootstrapped sampling distribution and CI's are valid (!), is set an arbitrary threshold (e.g. the 95th percentile of the differences, or a case where $B_k>1.5.A_k$, or $B_k-A_k>t_0$ with $t_0$ a value which would depend on the context of your measurements, etc.) and look for paired differences which exceed this threshold. Now these values would not be "statistically significant", but they would be "extreme", and possibly "practically significant" in the context of your study.

Note: to avoid such confusion about what the "Wilcoxon test" is, I personally always call the one the WSRt, and the other the MWUt (and never use the Wilcoxon sum rank test designation: it is totally correct, but prone to confusions, such as in this question).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.