Huge sample sizes, tests, and deviation of assumptions?

Question

I am performing Wilcoxon test (but in theory it can be any) and sometimes my sample sizes are huge. Even a single outlier may cause an extremely low p value.

I am not interested in small effects at all, my expected effects are huge.

How can I moderate impact of outliers in huge samples? I expect no more than 5 percents of points to be outliers and I can easily downsample to half of the sample size - the power will be enough. Or, in another words, how can I detect only effects of large size?

I am well aware of Bayesian ways, but here I need to get a p value in the end (I don't like it either, but that's what's expected from me).

In particular situation I was also thinking about simply reflecting all the points around the median and compare these reflected samples (so big or small values will be "balanced" from another side), but it also looks strange.

So, typically I compare 2 groups: one of size 10-1000, second is 100 times bigger. The distribution of groups is nothing special: reasonable variance, no guarantees on symmetry or family, equal in both groups if no outliers occur. If the groups are different, they are different by more than 4 SDs so it can be seen visually. However in the larger group there are "outliers". These outliers are relevant to experiment and don't occur due to technical error, however, I can not predict the outliers, I may just explain it after I observe it. So I dont have an informative way to get rid of them. The amount of outliers - I said it is 5 percents - but in most of the cases it is 1 or 2 per thousands of data points, but having such sample sizes, even a single outliers leads to pvalues of 10**-5 and lower.

The simulation:

pvals <- rep(0, 10000)
for (i in 1:length(pvals)) {
  one_sample <- rnorm(40)
  two_sample <- c(rnorm(100), -2, -2.01, -2.001)
  pvals[i] <- wilcox.test(one_sample, two_sample, alternative = "greater", exact = F)$p.value  
}

hist(pvals, breaks=50)

I added just 3 outliers which are 2SD from the mean - and you can see, even for Wilcoxon test, which is claimed to be more "robust" to outliers, it breaks.

UPD: none of the strategies I suggested in the question worked.

UPD2: that's the actual piece of data I am working with.

It is microarray data and only God knows how it is distributed. The group for comparison is around 10 dots and is located slightly lower than the median of the depicted distribution (but visually - same cloud of dots). I am interested in visually noticeable difference. There are "outliers" in both directions, but what really kills the analysis - the extreme outliers on top. However I am not sure if it will become much better if I remove only extreme outliers ...

Another UPD: I am thinking about this problem in a broader sense. I want to say that my null hypothesis is everything of small effect size. How can I do it using frequentist tests? Bayesian, having family of distributions - easily. Frequentists' way - have no idea.

Answer 1

I don't think the problem here is really the Wilcoxon test or any test. It is rather what you (or your colleagues) expect from a p-value. A small simulation shows that unequal sample sizes and 5% probability of extreme outliers does not cause a problem per se. Let's assume for both sample the mixture distribution $0.95 \times N(0,1) + 0.05 \times N(4,1)$, i.e. 5% of all values are completely apart from the rest of the distribution.

pvals.ttest <- rep(0, 10000)
pvals.wilcox <- rep(0, 10000)

for (i in 1:length(pvals)) {
  n1 <- 20           
  u1 <- runif(n1)
  x1 <- rep(NA,n1)

  for(j in 1:n1){
    if(u1[j]<0.95){
      x1[j] = rnorm(n=1,mean=0,sd=1)
    }else{
      x1[j] = exp(rnorm(n=1,mean=4,sd=1))
    }
  }

  n2 <- 200           
  u2 <- runif(n2)
  x2 <- rep(NA,n2)

  for(j in 1:n2){
    if(u2[j]<0.95){
      x2[j] = rnorm(n=1,mean=0,sd=1)
    }else{
      x2[j] = rnorm(n=1,mean=4,sd=1)
    }
  }

  pvals.ttest[i] <- t.test(x1, x2)$p.value  
  pvals.wilcox[i] <- wilcox.test(x1, x2, exact=F)$p.value  
}

sum(pvals.ttest<0.05)/10000
# [1] 0.0487
sum(pvals.wilcox<0.05)/10000
# [1] 0.0534

Both t-test and Wilcoxon are reasonably close to 0.05 which is just what would be expected under the null hypothesis.

We can now introduce a bias so that outliers only occur in the larger sample (however, I do not really understand why this should be the case in reality). We just change the condition for the first sampling procedure to if(u1[j]<=1). Then we get

> sum(pvals.ttest<0.05)/10000
[1] 0.1234
> sum(pvals.wilcox<0.05)/10000
[1] 0.0564

We see that the t-test is very sensitive to the outliers, while Wilcoxon is considerably robust. However, since the distributions are not the same anymore, we usually actually want to have significant p-values, since the null hypothesis does not hold.

If you or your colleages are not interested in rejecting the null hypothesis due to 5% outliers, I would propose to establish rules to identify outliers and conduct a sensitivity analysis. For example, consider the most extreme 5% values in each sample as outliers, and perform your statistical test once with the full samples and once with the reduced samples. However, if in both samples the probability of outliers is the same, then you do not need to do this. Small p-values indeed indicate evidence against the null hypothesis then.

Answer 2

One measure to describe data is the trimmed mean. It seems to suit your situation almost perfectly: Some ouliers, the rest regarded as well behaved. You need to decide how much to trim but you have a clear idea of how many outliers to expect.

One problem appears to be the sometimes small number in the small group: size 10-1000 will often be enough for a sensible calculation of a 10% trimmed mean but when the sample size gets as low 10 that is probably questionable.

I am not aware of a test for the trimmed mean but I have no doubt that building a permutation test could be straightforward. (Bootstrapping being an alternative but again, with $n_{min}=10$ questionable.