In the accepted answer of THIS question, it is stated that "With such large sample sizes both tests will have high power to detect minor differences". On the other hand, a voted up comment in THIS question states that with a very small sample size, the test "will fail no matter how large the difference between the mean of the groups".

I myself have serval large and imbalanced datasets, each of which has observations from two groups (e.g. 700 vs 10000). Using Mann-Whitney U test to analyse differences in these datasets has revealed a statistically significant difference in most cases. On the light of the comments above, I would assume that, at least for some cases, such differences may still be minor.

Based on my very limited (and intuitive!) understanding, I would assume that increasing the significance level required (e.g. using 0.01 instead of 0.05) to reject the null hypothesis may counteract the problem above and help to reject it only when a relatively large difference exist. Does this even make sense?

In the accepted answer of THIS question, it is stated that "With such large sample sizes both tests will have high power to detect minor differences". On the other hand, a voted up comment in THIS question states that with a very small sample size, the test "will fail no matter how large the difference between the mean of the groups".

I myself have serval large and imbalanced datasets, each of which has observations from two groups (e.g. 700 vs 10000). Using Mann-Whitney U test to analyse differences in these datasets has revealed a statistically significant difference in most cases. On the light of the comments above, I would assume that, at least for some cases, such differences may still be minor.

Based on my very limited (and intuitive!) understanding, I would assume that increasing the significance level required (e.g. using 0.01 instead of 0.05) to reject the null hypothesis may counteract the problem above and help to reject it only when a relatively large difference exist. Does this even make sense?

In the accepted answer of THIS question, it is stated that "With such large sample sizes both tests will have high power to detect minor differences. On the other hand, a voted up comment in THIS question states that with a very small sample size, the test "will fail no matter how large the difference between the mean of the groups".

I myself have serval large and imbalanced datasets, each of which has observations from two groups (e.g. 700 vs 10000). Using Mann-Whitney U test to analyse differences in these datasets has revealed a statistically significant difference in most cases. On the light of the comments above, I would assume that such differences may still be minor.

Based on my very limited (and intuitive!) understanding, I would assume that increasing the significance level required (e.g. using 0.01 instead of 0.05) to reject the null hypothesis may counteract the problem above and help to reject it only when a relatively large difference exist. Does this even make sense?

In the accepted answer of THIS question, it is stated that "With such large sample sizes both tests will have high power to detect minor differences". On the other hand, a voted up comment in THIS question states that with a very small sample size, the test "will fail no matter how large the difference between the mean of the groups".

I myself have serval large and imbalanced datasets, each of which has observations from two groups (e.g. 700 vs 10000). Using Mann-Whitney U test to analyse differences in these datasets has revealed a statistically significant difference in most cases. On the light of the comments above, I would assume that, at least for some cases, such differences may still be minor.

Based on my very limited (and intuitive!) understanding, I would assume that increasing the significance level required (e.g. using 0.01 instead of 0.05) to reject the null hypothesis may counteract the problem above and help to reject it only when a relatively large difference exist. Does this even make sense?