So I've been dealing with 2 sample hypothesis testing with very large samples (around 20,000s each). Whenever I test for the equality of distribution I always reject the null hypothesis, even though they aren't as different. I completely understand why this happens with large samples.
The advice around here is to use measureS of the effect size to assure that although the data comes from different distributions they aren't as different. The most recommended measure is cohens d $$d = \frac{\bar{x}-\bar{y}}{s_{\mbox{p}}}.$$ I think this measure is not that good because it only compare the standarized difference of means.
I thought that maybe the Wasserstein distance or the Kolmogorov-Smirnov statistic can be good measures of the effect size between the two distributions. Are they?
EDIT: A closer look:
I have a dataset with the prices of avocados from 2016-present. The dataset can be obtained here by doing some basic web scrapping. The data set has 81,867 observations. The thing with avocados is that they have different sizes (32, 36, 40 and 48), and I want to prove that the price of the avocados with sizes 32, 36, 40 and 48 are practically the same. The size of each subsample is $n_{32} = 10520, n_{36} = 13935, n_{40}= 13172$, and $n_{48}=24337$, and the following images shows a violinplot of the data and ECDF:
So I did a permutation test; with $H_0: \text{All distributions are the same}$. The procedure I used follows like this
- Calculate $t = \max\{D_{i,j}| \text{for } i,j = 32,36,40,48\}$ and $D_{i,j}$ is the Kolmogorov-Sminorv statistic for the original data.
- Permute the data
- From the permuted data get 4 samples without replacement with sizes $n_{32}, n_{36}, n_{40}$, and $n_{48}$
- Calculate $t_{\mbox{permuted}} = \max\{D_{i,j}| \text{for } i,j = 32,36,40,48\}$ and $D_{i,j}$ is the Kolmogorov-Sminorv statistic for the permutated samples. In words the maximium of the pairwise Kolmogorov-Smirnov statistic.
- Save $t_{\mbox{permuted}}$ and repeat 1000 times.
The next image shows the histogram of $t_{\mbox{permuted}}$ along with $t$.
So it's clear that null hypothesis is rejected. I get it that this happens because how large the samples are. After some investigation on this site the consensus was to study the effect size. The most recommended measure I encounter was cohens d $$d = \frac{\bar{x}-\bar{y}}{s_{\mbox{p}}}.$$ I think this measure is not that good because it only compare the standarized difference of means. So I thought that maybe the Wasserstein distance or the Kolmogorov-Smirnov statistic. The next table shows pairwise comparison between size and the cohen d, Wasserstein distance and the Kolmogorov-Smirnov statistic. My question is:
Can this measures provide enough information to assure that the effect size between differente sizes is small?