2
$\begingroup$

I generate two columns of length 343180 with random integer values between 0 and 290 and run sklearn's chi2-test of dependence. One would expect that the null hypothesis (independence) is accepted with a high probability, but actually I get a test score of approx. 15423 and a p-value of 0.

import numpy as np
from sklearn.feature_selection import chi2

X = np.transpose([[np.random.randint(0, 291) for i in range(0, 343180)]])
y = np.asarray([np.random.randint(0, 291) for i in range(0, 343180)])

print(X.shape)
# output: (34318, 1)

print(y.shape)
# output: (34318,)

chi2(X, y)
# output: (array([15423.73497325]), array([0.]))
# which means: p-value = 0.

Does this has to do with the limits of pseudo random number generation? Or do I misunderstand the concept of a chi2-test? Does the chi2-test, as implemented in sklearn, expect a certain type of distribution of the tested features, and not just an arbitrary discrete distribution?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ I can tell you now that the chi2 test is not doing what you would think it should be doing. I couldn't reproduce its output following a proper chi2 test for independence algo. The documentation is terrible, so it's impossible to say what algorithm they're implementing here without step-by-step debugging their code $\endgroup$
    – Aksakal
    Commented Dec 30, 2020 at 21:43
  • $\begingroup$ @Aksakal Re the rule of thumb for $n\times k$ tables (that expectations should be 5 or larger): with $n=k=291,$ even when the mean expectation per cell is just $1/2$ (!), the chi-squared statistic closely follows a chi-squared distribution under the null hypothesis. (Think of the CLT.) $\endgroup$
    – whuber ♦
    Commented Dec 30, 2020 at 21:49
  • 1
    $\begingroup$ @whuber at this point the problem is that I have no idea what is this function doing. i know for one that it won't be able to use the fact that expected frequencies of all cells are equal. yet even taking that into account it's still not clear WTH it is calculating. i dont have spare time to kill to step through its code $\endgroup$
    – Aksakal
    Commented Dec 30, 2020 at 23:07

2 Answers 2

Reset to default
2
$\begingroup$

I think you are right to doubt the output. I tried to replicate the result by implementing the Chi-square test as it is supposed to work for categorical data, e.g. see this link. I don't get the result that comes from sklearn.feature_selection.chi2, here's my code:and outputs:

import numpy as np
from scipy.stats import chi2
from sklearn import feature_selection 
np.random.seed(1)

c = 291
n = 343180
X = np.random.randint(c,size=(n,1))
y = np.random.randint(c,size=(n,1))

print('chi2 test stat, pval:',feature_selection.chi2(X, y))

# assuming classes are in rows, calculate expected for rows
p_class = np.zeros((c,1), dtype=np.double)
for yi in y:
  p_class[yi] = p_class[yi] + 1
p_class = p_class / n

n_feat = np.zeros((c,1))
for x in X:
  n_feat[x] = n_feat[x] + 1

exp = p_class @ n_feat.T

# feature levels are columns  
obs = np.zeros((c,c))
for i in np.arange(n):
  obs[y[i],X[i]] = obs[y[i],X[i]] + 1


#print(exp)
#print(obs)

stat = np.sum((obs-exp)**2/exp)
dof = (c-1)**2
chi2crit = chi2.ppf(1-0.05,df=dof)
pval = 1-chi2.cdf(stat,df=dof)
print('my stat,pval,crit(5%):',stat,pval,chi2crit)


chi2 test stat, pval: (array([13790.24430418]), array([0.]))
my stat,pval,crit(5%): 84025.71482666291 0.5712498907190868 84775.72566291611
Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ The rule of thumb doesn't apply to such huge tables. I believe four observations per cell is more than enough for the chi-squared distribution to approximate the actual distribution of the chi-squared statistic very well. $\endgroup$
    – whuber ♦
    Commented Dec 30, 2020 at 20:20
  • $\begingroup$ Aksakal, thanks so much for your implementation and confirmation of my doubts. I went through your code and this is exactly the algorithm that I learned in my class. I think the only minor mistake in your code is where you calculate "chi2crit" and "pval", there it should be "df=(c-1)**2", since this is the correct degree of freedom of the $\chi^2$ distribution. Then, using the same seed of 0, i get a p-value of approx. 0.5340, by the way exactly the same as scipy.stats.chi2_contingency(obs) yields. This p-value sounds reasonable, I don't understand why sk-learns p-value goes to 0 with large n. $\endgroup$
    – S. M. Roch
    Commented Dec 31, 2020 at 14:49
  • 1
    $\begingroup$ @S.M.Roch corrected degrees of freedom, forgot about the row and column sums used. the problem is in sklearn method's Chi-square stats, it's too low. p-value is an issue too, of course, clearly they use different degrees of freedom. I don't think what they do is wrong, it's just not what we think it's supposed to do - the issue must be a terrible documentation, and not a bug in the code. we don't know which algo they're implementing $\endgroup$
    – Aksakal
    Commented Dec 31, 2020 at 18:49
2
$\begingroup$

Three issues:

  1. Software engineering: Avoid 'magic numbers' in the code. Define the constants in the code and re-use them. That makes it easier to change the code to run with different values.

  2. Self-education: Have you tried changing any of your 'magic numbers'?

  3. Statistics: $\chi^2$-test checks for differences in ratios (relative frequencies). It would be quite a surprise if the ratios of so many random numbers would be the same.

So, to give you an answer by addressing all three points:

import numpy as np
from sklearn.feature_selection import chi2

U = 291
L = 0
N = 2 # used to be 343180
X = np.transpose([[np.random.randint(L, U) for i in range(0, N)]])
y = np.asarray([np.random.randint(L, U) for i in range(0, N)])

print(X.shape)
# output: (2, 1)

print(y.shape)
# output: (2,)

chi2(X, y)
# output: (array([7.42424242]), array([0.00643509]))
# which means: p-value < 0.007.
```
Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ "It would be quite a surprise if the ratios of so many numbers would be the same". This goes into the right direction probably, but still doesn't really answer my doubts. From what I understand $\chi^2$ tests $H_0: \{\textrm{Distribution of X and y are independent}\}$ against the alternative $H_1: \{\textrm{Distribution of X and y are dependent}\}$, and if the resulting p-value is very low that means that it is highly unlikely to produce such an "extreme" observation when assuming $H_0$. How can it depend then on the sample size? Actually, usually I assume the more samples the preciser it is. $\endgroup$
    – S. M. Roch
    Commented Dec 30, 2020 at 11:19
  • $\begingroup$ No, $H_0$ is "that there are no differences between the classes in the population" (en.wikipedia.org/wiki/Chi-squared_test). $\endgroup$
    – Igor F.
    Commented Dec 30, 2020 at 17:05
  • $\begingroup$ Thanks for pointing that out. There are several versions of $\chi^2$-tests. So your assumption is that sklearns test is a test of equal distributions? In that case, the p-value of 0 would make so much sense, because they are indeed equally (i.e. uniformly) distributed. But then it was not a test of dependency (as stated in the doc). Moreover, in the machine learning context of sklearn, X is a matrix with feature columns and y is a target variable. The columns in X and the y vector can have completely different encodings, and simply a test for same distribution wouldn't be so helpful, right? $\endgroup$
    – S. M. Roch
    Commented Dec 31, 2020 at 14:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.