12
$\begingroup$

I'm having trouble to understand how to compare 2 sets of data by their distribution .

For Example,
how can I understand that column X100 has the same distribution as column Y1?

enter image description here

enter image description here

Also, is there a way to express the distribution comparison of all columns to all columns?

I'm a machine learning developer using python, and this is a part of a classification problem I'm working on.

Would appreciate any help.. tnx :)

$\endgroup$

2 Answers 2

20
$\begingroup$

You can compare distribution of the two columns using two-sample Kolmogorov-Smirnov test, it is included in the scipy.stats: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ks_2samp.html

From the stackoverflow topic:

from scipy.stats import ks_2samp
import numpy as np

np.random.seed(123456)
x = np.random.normal(0, 1, 1000)
y = np.random.normal(0, 1, 1000)
z = np.random.normal(1.1, 0.9, 1000)

>>> ks_2samp(x, y)
Ks_2sampResult(statistic=0.022999999999999909, pvalue=0.95189016804849647)
>>> ks_2samp(x, z)
Ks_2sampResult(statistic=0.41800000000000004, pvalue=3.7081494119242173e-77)

Under the null hypothesis the two distributions are identical. If the K-S statistic is small or the p-value is high (greater than the significance level, say 5%), then we cannot reject the hypothesis that the distributions of the two samples are the same. Conversely, we can reject the null hypothesis if the p-value is low.

$\endgroup$
2
  • $\begingroup$ which quantity to check specifically? statistic or p-value? sometimes I get KstestResult(statistic=0.291, pvalue=0.3095011576244957). Here, statistic is not too high and p-value is low. Should I reject null hypothesis based on p-value or accept based on statistic $\endgroup$
    – Vinay
    Jun 30, 2022 at 13:37
  • 1
    $\begingroup$ @Vinay I worded it wrongly there, it's not an exclusive or. You look at p-value to judge how extreme the statistic (which is the "distance" between two distributions here) is. So if p-value is less than 0.05 it means you reject the null hypothesis. $\endgroup$ Jun 30, 2022 at 16:37
1
$\begingroup$

To compare all columns to all columns, maybe you can create a response label column with "1" as data from dataset 1 and "0" as data from dataset 2. You can build a classification task over this response label using all columns in the combined dataset. If you can get a good AUC score, then the data is separable and the dataset 1 and dataset 2 are probably from two different distributions.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.