5
$\begingroup$

Assume I want to do some, say, churn analysis on a dataset. Decision Trees, for instance, are relatively robust to skewed distributions in the (numerical) features, but are rather poor on imbalanced targets (as is standard).

My question is: what about imbalanced categorical features e.g., the feature sex having 10% male and 90% females? Are classifiers such as Random Forest, Logistic Regression, and SVM robust against imbalanced categorical features? If not, are there some ways to overcome this by e.g., under/oversample features or...?

Clearly, if we have a small dataset it is a problem, but generally speaking, I am curious.

$\endgroup$
4
  • $\begingroup$ I believe Yes but would like to hear of any systematic research into the question. $\endgroup$
    – rolando2
    Commented May 23, 2020 at 13:30
  • $\begingroup$ That is exactly my thoughts $\endgroup$
    – CutePoison
    Commented May 23, 2020 at 13:32
  • $\begingroup$ A related research area is optimal experimental design for (e.g.) logistic regressions. One way to measure the a priori effect of covariate imbalance would be to compare a heavily imbalanced design of size N to the optimal design of size N (given an optimality criterion, a parameter vector, etc. etc.). I think this is where the systematic research lies. $\endgroup$
    – JTH
    Commented May 25, 2020 at 4:04
  • $\begingroup$ Agreed, but if there was some analytical proof/papers regarding it, that would be great. I might set down and to some testing my self instead. I really struggle to find anything about it, so I doubt it is a problem which is often encountered (thus is maybe not problem?) $\endgroup$
    – CutePoison
    Commented May 25, 2020 at 7:08

1 Answer 1

1
$\begingroup$

It might be that, with the feature having so little diversity, the usual idea in regression and supervised learning problems to use variation in the features to explain variation in the outcome breaks down. However, it might be that a feature that occasionally takes a value is quite informative. You give an example where $90\%$ of the observations are female and the remaining $10\%$ male. It might be that subject sex is a strong predictor of the outcome and is screaming at you that a particular outcome is likely, such as using the sex of a cow to predict if it can produce milk. Thus, I would hesitate to exclude a feature just because it lacks balance in its categories.

$\endgroup$
1
  • 1
    $\begingroup$ (+1). A subsequent question might be "How likely is it that the predictor is screaming at me just out of (bad) luck?". $\endgroup$
    – J-J-J
    Commented Apr 4, 2023 at 21:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.