What's the correct approach to measure "correlation" in binary problems?

Question

I know that one of the principal steps within a ML problem is look at the data to gain some insights about it, so to do so, something that could help is exploring the correlation of our features. For that, I know you can use Pearson's correlation coefficient for linear correlation or some other metrics if you're interested in nonlinear dependencies, like Spearman's coefficient.

The problem I've found is that, that is fine when all my features are continous, but what happens if they are not?

If I have in my problem some features that are binary, for example, or even my target variable is binary (a classification problem), which is the correct approach to measure the "correlation" between them and the others? (I think that isn't the appropiate term, but "association", although I'm not sure). The goal is the same, measure which features depends on each other and how much they do it.

And, to complete the question, what happens if the features are not binary but categorical (more than 2 categories)? Is there some specific metric like Pearson's coefficient for that situation, or the standard approach is to encoding them numerically and then apply wichever the solution for the binary case is?

I've look for the answer of these questions before posting them here, but what I've found is several people saying way different methods to do this, so instead of picking one of those and that's it I would like to know first what's the correct approach with this kind of problems.

Thank you!

Answer 1

Assumption: You are developing in Python.

In the true StackOverflow fashion, I suggest not using correlation and using Mutual Information instead, as it captures both linear and non-linear relationships and is more general than correlation coefficients. Note that, Mutual information calculation is comparatively computationally intensive.

• Mutual Information:
  • Library: scikit-learn
    • mutual_info_score: Used for measuring mutual information between two categorical variables.
    • mutual_info_regression: Used for measuring mutual information between a continuous target variable and one or more continuous or categorical predictor variables, typically in the context of regression problems.
    • mutual_info_classif: Used for measuring mutual information between a categorical target variable and one or more continuous or categorical predictor variables, typically in the context of classification problems.

Nevertheless, here are some methods for detecting correlation/association/dependence between variables:

• Binary & Continuous: Point-biserial correlation coefficient -- a special case of Pearson's correlation coefficient, which measures the linear relationship's strength and direction.

  • Library: SciPy (pointbiserialr)

• Binary & Binary: Phi coefficient or Cramér's V -- based on the chi-squared statistic and measures the association between them.

  • Library: scikit-learn (matthews_corrcoef) and custom function for Cramér's V with pandas

• Categorical & Continuous: ANOVA -- tests if their mean is significantly different across the categories.

  • Library: SciPy (f_oneway)

• Categorical & Categorical: Chi-squared test or Cramér's V -- measure of association between them.

  • Library: SciPy (chi2_contingency) and custom function for Cramér's V with pandas

Answer 2

Pearson's correlation, at least its magnitude, between two numerical variables is equivalent to taking the square root of the $R^2$ from an ordinary least squares (OLS) linear regression regression of one variable on the other. If you can develop a regression with categorical variables and calculate something like $R^2$ for that regression, you are set.

Fortunately, such regressions and metrics do exist. Logistic regression (binary outcome) and multinomial logistic regression ($3+$ categories in the outcome) are reasonable analogues of OLS linear regression regression with numerical variables. Further, both models can use categorical or numerical variables. From such regressions, pseudo $R^2$ values can be calculated. My favorite from the link would be the McFadden $R^2$ that uses the likelihood (in the technical sense) of the model compared to the likelihood of a model that always predicts the pooled (prior) probability. You could then take the square root of the McFadden $R^2$ to quantify the strength of the relationship between your feature and categorical outcome.

Note, however, that univariate variable screening presents problems.