Wasserstein distance and Kolmogorov-Smirnov statistic as measures of effect size

Question

So I've been dealing with 2 sample hypothesis testing with very large samples (around 20,000s each). Whenever I test for the equality of distribution I always reject the null hypothesis, even though they aren't as different. I completely understand why this happens with large samples.

The advice around here is to use measureS of the effect size to assure that although the data comes from different distributions they aren't as different. The most recommended measure is cohens d $$d = \frac{\bar{x}-\bar{y}}{s_{\mbox{p}}}.$$ I think this measure is not that good because it only compare the standarized difference of means.

I thought that maybe the Wasserstein distance or the Kolmogorov-Smirnov statistic can be good measures of the effect size between the two distributions. Are they?

EDIT: A closer look:

I have a dataset with the prices of avocados from 2016-present. The dataset can be obtained here by doing some basic web scrapping. The data set has 81,867 observations. The thing with avocados is that they have different sizes (32, 36, 40 and 48), and I want to prove that the price of the avocados with sizes 32, 36, 40 and 48 are practically the same. The size of each subsample is $n_{32} = 10520, n_{36} = 13935, n_{40}= 13172$, and $n_{48}=24337$, and the following images shows a violinplot of the data and ECDF:

So I did a permutation test; with $H_0: \text{All distributions are the same}$. The procedure I used follows like this

0. Calculate $t = \max\{D_{i,j}| \text{for } i,j = 32,36,40,48\}$ and $D_{i,j}$ is the Kolmogorov-Sminorv statistic for the original data.
1. Permute the data
2. From the permuted data get 4 samples without replacement with sizes $n_{32}, n_{36}, n_{40}$, and $n_{48}$
3. Calculate $t_{\mbox{permuted}} = \max\{D_{i,j}| \text{for } i,j = 32,36,40,48\}$ and $D_{i,j}$ is the Kolmogorov-Sminorv statistic for the permutated samples. In words the maximium of the pairwise Kolmogorov-Smirnov statistic.
4. Save $t_{\mbox{permuted}}$ and repeat 1000 times.

The next image shows the histogram of $t_{\mbox{permuted}}$ along with $t$.

So it's clear that null hypothesis is rejected. I get it that this happens because how large the samples are. After some investigation on this site the consensus was to study the effect size. The most recommended measure I encounter was cohens d $$d = \frac{\bar{x}-\bar{y}}{s_{\mbox{p}}}.$$ I think this measure is not that good because it only compare the standarized difference of means. So I thought that maybe the Wasserstein distance or the Kolmogorov-Smirnov statistic. The next table shows pairwise comparison between size and the cohen d, Wasserstein distance and the Kolmogorov-Smirnov statistic. My question is:

Can this measures provide enough information to assure that the effect size between differente sizes is small?

Answer 1

So you are comparing prices, and your violin plots doesn't show much of a difference. But with your large sample sizes, you could use a method that investigates where are the differences, exactly? One such method is relative distribution plots, it could be interesting with your data to see such plots!

For an example and more information see What are good data visualization techniques to compare distributions?. With your data it could make sense to use 32s as baseline group, making for three plots.

Generally, with larger sample sizes it makes sense to make more detailed comparisons, and it could be interesting for your clients.