For a problem at work I have a continuous variable between 0 and 1. Roughly normally distributed but not exactly. I have two groups of customers and want to compare them if one group is different from the other wrt to this one variable.
I plotted their distributions and they almost perfectly overlap. That said, when running a t-test of Mann Whitney (b/c not exactly normal distribution), there is a statistically significant difference. How is that possible?
For obvious reasons can't post my data, so here's simulated data that shows a similar phenomenon:
import numpy as np
from scipy.stats import ttest_ind
a = np.random.normal(loc=0.41, scale=0.2, size=10000)
b = np.random.normal(loc=0.4, scale=0.2, size=10000)
print(ttest_ind(a,b)) # --> returns a tiny p_value < 0.01
I'm fairly embarrassed I need to ask this, but please help. Thanks.