I am using an automated variable selection technique to find covariates in a regression model. Notwithstanding the issues with automated variable selection, the procedure has returned two categorical covariates, RACE and LOCATION. RACE has five categories (1,2,3,4,5), and LOCATION has two (A and B). Part of the issue is that in location A, most of the RACE observations fall into a specific category, so it seems including both covariates is redundant...?

What is a typical procedure to deal with this? Would I want to combine categories somehow?


A simple solution is to dichotomize all your categorical features before running the automated variable selection process. For instance, you can create five dummy (binary) features for RACE:

  1. RACE_1 = 1 when RACE = 1, else 0
  2. RACE_2 = 1 when RACE = 2, else 0
  3. ...

The automated variable selection process should have a subroutine that removes heavily redundant (correlated) features. This is relatively easy to do once you have all numeric features (continuous, and binary) instead of a mix of numeric and character fields.

One approach to remove such redundancy is to study pairwise correlations between all of your input features and identify heavily correlated pairs. From each such pair, discard the field that has a lower correlation with the dependent variable.

In addition (or alternatively, if you prefer), you can perform multicollinearity diagnostics to remove fields that are heavily correlated with other fields.

  • $\begingroup$ But is it necessary to re-do automated variable selection? In other words, is it also plausible to just create a new categorical variable that combines RACE and LOCATION i.e. If RACE= 1 and LOCATION =A, RACELOC = 1, etc? $\endgroup$ – user85727 Mar 21 '16 at 23:04
  • $\begingroup$ You could create those new variables by combining RACE and LOCATION. However, the new variables should be scrutinized by the automated variable selection process, otherwise they would be getting a free pass. $\endgroup$ – Vishal Mar 22 '16 at 13:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.