0
$\begingroup$

I have the following table as pandas dataframe with features feat1 and feat2:

testframe = pd.DataFrame(columns = ['feat1', 'feat2'])
testframe['feat1'] = [1,0,1,0,1,0,1,1,0,1]
testframe['feat2'] = [1,0,1,0,0,0,1,1,0,0]

where the index is the number of observation (e.g people).

I want to find out, if there are any correlation between feat1 and feat2. I thought about chi2, but I am not quite sure if it is applicable in this case. My real observation has the shape (43,2), so it is quite small I would say. Are there any tests for correlation (or independency) you can recommend? If so, how would be the implementation in python? I am happy for every hint on this! Thanks!

$\endgroup$
  • $\begingroup$ A chi-squared test in this 2 by 2 table should work fine. $\endgroup$ – whuber May 15 at 20:29
1
$\begingroup$

You can test for association between the two columns, so you build a contingency table like:

tab = pd.crosstab(testframe['feat1'],testframe['feat2'])
tab

feat2   0   1
feat1       
   0    4   0
   1    2   4

If there is correlation between you two variables, the counts will populate the diagonal. Like @whuber suggested, you can first try chi-square test:

import pandas as pd
from scipy.stats import fisher_exact, chi2_contingency
chi2_contingency(tab)

(2.100694444444445,
 0.14723225536366139,
 1,
 array([[2.4, 1.6],
        [3.6, 2.4]]))

The first value is the chi-square statistic, 2nd is the p-value, degree of freedom and last is the table of expected values. If none of your expected counts is the array is < 5, you can use the chi-square.

In the situation some of the expected has <5, use a fisher test:

fisher_exact(tab)
(inf, 0.0761904761904761)

The first value is odds ratio (inf because you off diagonal gives 0), and the second is the p-value.

The manual for the test:

https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.chi2_contingency.html#scipy.stats.chi2_contingency

https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.fisher_exact.html

| cite | improve this answer | |
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.