2
$\begingroup$

Let's say I have four sets of data pairs: height and salary of some people from Cambodia, Vietnam, Poland and Italy, with the number of data points are different for each of these countries.

My hypothesis is something along the lines of: the salary is not too different between Cambodia and Vietnam or between Poland and Italy, but there's a large distance between the first two and the last two. For the height of the people, I expect all four countries to have the same degree of difference.

What is the correct statistical test to quantify this 'distance' between the distributions? I've used the Kolmogorov-Smirnov test before, but if I understand the situation correctly, that is only to test if the two datasets are from the exact same distribution - which is not true in my case.

I don't know all that much about entropy measures, but I think the data has to have the same size (this could be solved somehow by binning or random selection I believe) but also normalized to be between 0 and 1. Wouldn't this normalization destroy the fact that the datasets are not on the same scale?

I think what I am looking for is something similar to Pearson's R value, but a non-linear extension of it I think. Is there such a thing that's easy to calculate (especially in Python)?

Just for reference, the height histogram for Poland and Cambodia would look as such (ignore the fact that the numbers are meaningless, of course it's not real data):

enter image description here

enter image description here

Note that even the spread of the data is quite different. The salaries of the two countries would be:

enter image description here

enter image description here

Is there any meaningful metric that could measure if the salaries are more similar between these countries than the heights?

$\endgroup$
0

1 Answer 1

4
$\begingroup$

The Kolmogorov-Smirnov test statistic is in fact a distance measure between distributions. There are alternatives, e.g., https://en.wikipedia.org/wiki/Wasserstein_metric https://en.wikipedia.org/wiki/Hellinger_distance

You could also define a distance that only compares means, or means and variances. Obviously such a distance would not use all information, but ultimately there is no unique meaning to the concept of a "distance" and the researcher has to decide what aspects of the distribution are relevant to the problem in hand.

$\endgroup$
2
  • $\begingroup$ If I do a 'traditional' two-way Kolmogorov-Smirnov test on any of these distributions, the p-value turns out to be about 10^-6 or so. This, to me is not that surprising since the way I've learnt this method is that the p value shows the probability that the two distributions are the same, which is of course not true. Can I use this 10^-6 as a distance, is it a meaningful number? $\endgroup$
    – user337493
    Oct 11, 2021 at 7:12
  • 2
    $\begingroup$ You can use the value of the test statistic, not the p-value. $\endgroup$ Oct 11, 2021 at 12:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy