4
$\begingroup$

So I've been dealing with 2 sample hypothesis testing with very large samples (around 20,000s each). Whenever I test for the equality of distribution I always reject the null hypothesis, even though they aren't as different. I completely understand why this happens with large samples.

The advice around here is to use measureS of the effect size to assure that although the data comes from different distributions they aren't as different. The most recommended measure is cohens d $$d = \frac{\bar{x}-\bar{y}}{s_{\mbox{p}}}.$$ I think this measure is not that good because it only compare the standarized difference of means.

I thought that maybe the Wasserstein distance or the Kolmogorov-Smirnov statistic can be good measures of the effect size between the two distributions. Are they?

EDIT: A closer look:

I have a dataset with the prices of avocados from 2016-present. The dataset can be obtained here by doing some basic web scrapping. The data set has 81,867 observations. The thing with avocados is that they have different sizes (32, 36, 40 and 48), and I want to prove that the price of the avocados with sizes 32, 36, 40 and 48 are practically the same. The size of each subsample is $n_{32} = 10520, n_{36} = 13935, n_{40}= 13172$, and $n_{48}=24337$, and the following images shows a violinplot of the data and ECDF:

enter image description here

enter image description here

So I did a permutation test; with $H_0: \text{All distributions are the same}$. The procedure I used follows like this

  1. Calculate $t = \max\{D_{i,j}| \text{for } i,j = 32,36,40,48\}$ and $D_{i,j}$ is the Kolmogorov-Sminorv statistic for the original data.
  2. Permute the data
  3. From the permuted data get 4 samples without replacement with sizes $n_{32}, n_{36}, n_{40}$, and $n_{48}$
  4. Calculate $t_{\mbox{permuted}} = \max\{D_{i,j}| \text{for } i,j = 32,36,40,48\}$ and $D_{i,j}$ is the Kolmogorov-Sminorv statistic for the permutated samples. In words the maximium of the pairwise Kolmogorov-Smirnov statistic.
  5. Save $t_{\mbox{permuted}}$ and repeat 1000 times.

The next image shows the histogram of $t_{\mbox{permuted}}$ along with $t$.

enter image description here

So it's clear that null hypothesis is rejected. I get it that this happens because how large the samples are. After some investigation on this site the consensus was to study the effect size. The most recommended measure I encounter was cohens d $$d = \frac{\bar{x}-\bar{y}}{s_{\mbox{p}}}.$$ I think this measure is not that good because it only compare the standarized difference of means. So I thought that maybe the Wasserstein distance or the Kolmogorov-Smirnov statistic. The next table shows pairwise comparison between size and the cohen d, Wasserstein distance and the Kolmogorov-Smirnov statistic. My question is:

Can this measures provide enough information to assure that the effect size between differente sizes is small?

$$

$\endgroup$
3
  • 1
    $\begingroup$ The thing is that different methods measure different aspects of the distance between distributions. You have to decide what aspects you are interested in, which is connected to the ultimate goal as mentioned by @kjetilbhalvorsen. Kolmogorov and Wasserstein can be used, but I doubt many will feel confident interpreting the resulting values, as specifying "effect sizes" is not very common for this problem. In fact your graphs give a pretty clear idea about how similar these distributions are. Why isn't this enough? $\endgroup$ Oct 21, 2021 at 17:30
  • $\begingroup$ By the way, your edited question now has some repetitions that you probably don't want. $\endgroup$ Oct 21, 2021 at 17:32
  • $\begingroup$ Definitely is enough and even more for the clients. I got really into hypothesis testing during this analysis specially to learn more on what to do when very large samples. I will take a look at the repetitions. Thanks $\endgroup$
    – Girigio
    Oct 21, 2021 at 17:38

1 Answer 1

1
$\begingroup$

So you are comparing prices, and your violin plots doesn't show much of a difference. But with your large sample sizes, you could use a method that investigates where are the differences, exactly? One such method is relative distribution plots, it could be interesting with your data to see such plots!

For an example and more information see What are good data visualization techniques to compare distributions?. With your data it could make sense to use 32s as baseline group, making for three plots.

Generally, with larger sample sizes it makes sense to make more detailed comparisons, and it could be interesting for your clients.

$\endgroup$
2
  • $\begingroup$ Defenitly will do that analysis. Thanks, now that I think of it I remember that we cover that topic intro stats class during my masters. $\endgroup$
    – Girigio
    Oct 21, 2021 at 18:30
  • 1
    $\begingroup$ When you have done that, please show us the results by answering your own question! $\endgroup$ Oct 21, 2021 at 18:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.