0
$\begingroup$

For a problem at work I have a continuous variable between 0 and 1. Roughly normally distributed but not exactly. I have two groups of customers and want to compare them if one group is different from the other wrt to this one variable.

I plotted their distributions and they almost perfectly overlap. That said, when running a t-test of Mann Whitney (b/c not exactly normal distribution), there is a statistically significant difference. How is that possible?

For obvious reasons can't post my data, so here's simulated data that shows a similar phenomenon:

import numpy as np
from scipy.stats import ttest_ind

a = np.random.normal(loc=0.41, scale=0.2, size=10000)
b = np.random.normal(loc=0.4, scale=0.2, size=10000)

print(ttest_ind(a,b)) # --> returns a tiny p_value < 0.01

enter image description here

I'm fairly embarrassed I need to ask this, but please help. Thanks.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Regardless of whether you're using a standard independent t-test (as your code suggests) or a Mann Whitney (as you say in the description), be careful with significance and large samples.

Tests of statistical significance can reveal "significant" effects that are not particularly meaningful, especially with larger samples (e.g., How to perform t-test with huge samples?).

Also, independent t-tests are reasonably robust (How robust is the independent samples t-test when the distributions of the samples are non-normal?), even with samples less than 10K.

This might be a duplicate question, since it overlaps with some other topics.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for your reply. In the code I used a t-test b/c I drew from normal distributions for the example. In practice I used MW b/c my actual data has a few kinks so not exactly normally distributed. I also confirmed this with shuffling (merged the two sets to one, shuffled them, split to two, calculated the diff in means, and repeated that 1000 times). Am I wrong in thinking that if there's a statistically significant difference in means there should be very small, if any, overlapping area between the two curves? See this illustration pls $\endgroup$
    – Optimesh
    Apr 28, 2021 at 21:33
  • $\begingroup$ See the first link or the link in this comment for significance tests in large samples. The image you posted is oversimplified. Much of the discussion has to do with the meaningfulness of the finding - a small effect may be statistically significant in large samples (stats.stackexchange.com/questions/2516/…). I don't know what you mean by confirming non-normality by combining two groups of data and splitting them apart. Small deviations of non-normality may not be as important as you think - please see the second link I posted. $\endgroup$
    – pep
    Apr 28, 2021 at 21:53
  • $\begingroup$ sorry, I meant that I confirmed the statistically significant difference with a permutation test. My original question remains unanswered though, I believe. $\endgroup$
    – Optimesh
    May 1, 2021 at 13:59
  • $\begingroup$ tl;dr - statistical significance can occur with large samples, even if the effect is very small. This happened in your normality test and test of between-groups effects. Try reducing the sample size to 100 for each group and you'll get a different p-value every time. You'll also see that the t value can be positive or negative (either random group of 100 could be larger or smaller). P-values are complicated! Another excellent resource: coursera.org/learn/statistical-inferences. Advice: try thinking about effect size, rather than statistical significance. Hope this helps! $\endgroup$
    – pep
    May 9, 2021 at 15:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.