Can increasing the significance level help to avoid detecting minor differences in large datasets?

Question

In the accepted answer of THIS question, it is stated that "With such large sample sizes both tests will have high power to detect minor differences". On the other hand, a voted up comment in THIS question states that with a very small sample size, the test "will fail no matter how large the difference between the mean of the groups".

I myself have serval large and imbalanced datasets, each of which has observations from two groups (e.g. 700 vs 10000). Using Mann-Whitney U test to analyse differences in these datasets has revealed a statistically significant difference in most cases. On the light of the comments above, I would assume that, at least for some cases, such differences may still be minor.

Based on my very limited (and intuitive!) understanding, I would assume that increasing the significance level required (e.g. using 0.01 instead of 0.05) to reject the null hypothesis may counteract the problem above and help to reject it only when a relatively large difference exist. Does this even make sense?

Answer 1

It can make sense but based on your wish to detect only "non-minor" location differences, you have two much more natural and focussed options:

• You compute a confidence interval $I$ for the true location difference $\Delta$ and check if all its values are incompatible with what you call "minor difference". In formula notation, if $I \cap [\pm\delta]=\emptyset$, then you claim that the true difference $\Delta$ is non-minor. Here, $\delta > 0$ is a fixed value quantifying what you call "minor difference".

• You test two alternative hypotheses $H_1: \Delta > \delta$ and also $H_2: \Delta < -\delta$. If you can reject at least one of these, you claim that the true difference is non-minor.

Actually, the two approaches are identical.