6
$\begingroup$

I know that one of the principal steps within a ML problem is look at the data to gain some insights about it, so to do so, something that could help is exploring the correlation of our features. For that, I know you can use Pearson's correlation coefficient for linear correlation or some other metrics if you're interested in nonlinear dependencies, like Spearman's coefficient.

The problem I've found is that, that is fine when all my features are continous, but what happens if they are not?

If I have in my problem some features that are binary, for example, or even my target variable is binary (a classification problem), which is the correct approach to measure the "correlation" between them and the others? (I think that isn't the appropiate term, but "association", although I'm not sure). The goal is the same, measure which features depends on each other and how much they do it.

And, to complete the question, what happens if the features are not binary but categorical (more than 2 categories)? Is there some specific metric like Pearson's coefficient for that situation, or the standard approach is to encoding them numerically and then apply wichever the solution for the binary case is?

I've look for the answer of these questions before posting them here, but what I've found is several people saying way different methods to do this, so instead of picking one of those and that's it I would like to know first what's the correct approach with this kind of problems.

Thank you!

$\endgroup$
3
  • $\begingroup$ Cramér's V is a method that can be used to determine the correlation among the categorical variables. $\endgroup$
    – user344849
    Commented Dec 23, 2021 at 0:54
  • $\begingroup$ Ordinary Pearson correlation is useful for some purposes when the variables are binary. There is no "thou shalt not" here. $\endgroup$ Commented Dec 29, 2021 at 23:34
  • $\begingroup$ See also stats.stackexchange.com/questions/103801/… $\endgroup$
    – Nick Cox
    Commented Jul 5 at 15:57

2 Answers 2

3
$\begingroup$

Assumption: You are developing in Python.

In the true StackOverflow fashion, I suggest not using correlation and using Mutual Information instead, as it captures both linear and non-linear relationships and is more general than correlation coefficients. Note that, Mutual information calculation is comparatively computationally intensive.

  • Mutual Information:
    • Library: scikit-learn
      • mutual_info_score: Used for measuring mutual information between two categorical variables.
      • mutual_info_regression: Used for measuring mutual information between a continuous target variable and one or more continuous or categorical predictor variables, typically in the context of regression problems.
      • mutual_info_classif: Used for measuring mutual information between a categorical target variable and one or more continuous or categorical predictor variables, typically in the context of classification problems.

Nevertheless, here are some methods for detecting correlation/association/dependence between variables:

  • Binary & Continuous: Point-biserial correlation coefficient -- a special case of Pearson's correlation coefficient, which measures the linear relationship's strength and direction.

    • Library: SciPy (pointbiserialr)
  • Binary & Binary: Phi coefficient or Cramér's V -- based on the chi-squared statistic and measures the association between them.

  • Categorical & Continuous: ANOVA -- tests if their mean is significantly different across the categories.

    • Library: SciPy (f_oneway)
  • Categorical & Categorical: Chi-squared test or Cramér's V -- measure of association between them.

$\endgroup$
1
  • 1
    $\begingroup$ An upside to mutual information is that it captures all kinds of nonlinear relationships that correlation misses. A downside to mutual information is that it captures all kinds of nonlinear relationships that correlation misses and that might not be present in a model, leading someone to include a feature that lacks a strong linear relationship but only model linearly. // I am not convinced that nonlinear relationships between categorical variables can exist. Here, I show this for the binary case. Categorical features are a natural extension. $\endgroup$
    – Dave
    Commented Apr 24, 2023 at 18:15
0
$\begingroup$

Pearson's correlation, at least its magnitude, between two numerical variables is equivalent to taking the square root of the $R^2$ from an ordinary least squares (OLS) linear regression regression of one variable on the other. If you can develop a regression with categorical variables and calculate something like $R^2$ for that regression, you are set.

Fortunately, such regressions and metrics do exist. Logistic regression (binary outcome) and multinomial logistic regression ($3+$ categories in the outcome) are reasonable analogues of OLS linear regression regression with numerical variables. Further, both models can use categorical or numerical variables. From such regressions, pseudo $R^2$ values can be calculated. My favorite from the link would be the McFadden $R^2$ that uses the likelihood (in the technical sense) of the model compared to the likelihood of a model that always predicts the pooled (prior) probability. You could then take the square root of the McFadden $R^2$ to quantify the strength of the relationship between your feature and categorical outcome.

I challenge the idea that nonlinear relationships can exist in such a setting.

Note, however, that univariate variable screening presents problems.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.