4
$\begingroup$

Perhaps this is a very basic question, but I didn't find yet a simple solution for this simple problem:

I want to compare two samples (say X and Y) for a continuous variable which is non-normally distributed and test if X and Y are significantly different. The sample size of X is N=81 and Y is N=5110, so they are quite unbalanced. My first attempt was to use the Mann-Whitney (i.e. Wilcoxon Signed Rank test). However, I am bothered with this huge difference in sample sizes.

I thought that some kind of randomization or bootstrap method is a good alternative, but I am not sure if my approach makes sense. My idea was to get 1000 random samples of size 81 from Y and X and then use the Mann-Whitney to compare both distributions. The empirical p-value would be the proportion of tests with p-value < 0.05. I "R", I've implemented as follows:

X = data1 # sample size 81
Y = data2 # sample size 5510
R = 1000
alpha = numeric(R)

for(i in 1:R) {
    group1 = sample(X, replace=TRUE)
    group2 = sample(Y, size=81, replace=TRUE)
    alpha[i] = wilcox.test(group1, group2)$p.value
}

Empirical p-value would be the proportion of p-values < 0.05:

mean(alpha < 0.05)

Does this approach make sense? How can I do this hypothesis testing correctly?

$\endgroup$
  • 2
    $\begingroup$ "I am bothered with this huge difference in sample sizes" -- why is this bothersome to you? (If you feel a randomization test is okay, why is a randomization test on the ranks suddenly a problem?) $\endgroup$ – Glen_b -Reinstate Monica Mar 20 '14 at 0:46
  • $\begingroup$ Hi Glen, thanks for your comment. I am very newbie in statistics and I don't know to answer you, I just thought that such a large difference in sample sizes would be unfair... $\endgroup$ – gufranca Mar 20 '14 at 0:52
  • $\begingroup$ I'm not sure there's actually a problem; if you can clarify what sort of problem there might be, or ask the question in a different way (such as "Is there a problem with...?"), that might make it easier to respond to. $\endgroup$ – Glen_b -Reinstate Monica Mar 20 '14 at 2:38
  • $\begingroup$ You might also use a Bayesian estimation of difference via the BEST package $\endgroup$ – Ben Mar 20 '14 at 7:10
5
$\begingroup$

I am not a big expert on statistical testing, but the approach you are considering decidedly does not make sense. Imagine that the groups are indeed identical (i.e. null hypothesis is true). Then you will observe p<0.05 in exactly 5% of the cases, and e.g. p<0.01 in 1% of the cases (those would be false positives). So following your logic, you would reject the null.

I am not aware of any problems with Wilcoxon-Mann-Whitney test in case of different numbers of observations. So one option you have is to run the ranksum test as usual, without any further complications.

However, if you do feel concerned about the very different $N$, you can try a simple permutation test: pool both groups together (obtaining $81+5110=5191$ numbers) and randomly select $81$ values as group A and all the rest as group B. Then take the difference between the means (or medians) of A and B (let's call it $\mu$), and repeat this many many times. This will give you a distribution $p(\mu)$. At the same time for your actual groups X and Y you have some fixed empirical value of $\mu^*$. Now you can check if $\mu^*$ lies in the 95% percentile interval of $p(\mu)$. If it does not, you can reject the null with p<0.05.

$\endgroup$
1
$\begingroup$

Your approach does not make sense. The usual Wilcoxon-test will answer you with high power. Your approach looses this advantage. It may however be reasonable to be afrait of too much power, because even irrelevant differences will show up significant, which would in fact distract the scientist interested in a relevant qualitative statement of a test.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.