# Feature Selection with Categorical Variables: Multicollinearity and Statistical Significance

Building a logistic regression model with three categorical features and one continuous. For simplicity, let's say I have the following features and variables:

      Feature    |        Variable
========================================
Gender     |         Male
----------------------------------------
Weight     |  Weight (continuous)
----------------------------------------
Income Bucket |        0 - 50k
Income Bucket |       51k - 75k
Income Bucket |       76k - 100k
Income Bucket |       101k - 150k
----------------------------------------
Continent   |      North America
Continent   |      South America
Continent   |         Europe
Continent   |          Asia
Continent   |         Africa
Continent   |        Australia


Recall that I am using for the number of variables so there is no variable for Gender = Female, Income Bucket = 100k+, or Continent = Antarctica.

1. if I run this through a multicollinearity check and get VIFs for one categorical variable being higher than 10, does it make sense to remove it?

For example:

      Feature    |        Variable       |  VIF  |
=================================================
Gender     |         Male          |   2
-------------------------------------------------
Weight     |  Weight (continuous)  |   1
-------------------------------------------------
Income Bucket |        0 - 50k        |   3
Income Bucket |       51k - 75k       |   4
Income Bucket |       76k - 100k      |   11 <--
Income Bucket |       101k - 150k     |   1
-------------------------------------------------
Continent   |      North America    |   2
Continent   |      South America    |   3
Continent   |         Europe        |   4
Continent   |          Asia         |   1
Continent   |         Africa        |   1
Continent   |        Australia      |   2


Should I drop the categorical variable for Income Bucket = 76k- 100k and then proceed to further model selection?

1. Furthermore, does it make sense if one or more (but not all) have p-values are not significant?

For example:

      Feature    |        Variable       |  P-Value  |
=====================================================
Gender     |         Male          |   0.002
-----------------------------------------------------
Weight     |  Weight (continuous)  |   0.001
-----------------------------------------------------
Income Bucket |        0 - 50k        |    0.05
Income Bucket |       51k - 75k       |    1.65 <--
Income Bucket |       76k - 100k      |    0.03
Income Bucket |       101k - 150k     |   0.002
-----------------------------------------------------
Continent   |      North America    |    0.05
Continent   |      South America    |   0.001
Continent   |         Europe        |    11.2 <--
Continent   |          Asia         |    0.01
Continent   |         Africa        |    0.09
Continent   |        Australia      |    0.1


Here, does it make sense to drop Income Bucket = 51k - 75k and Continent = Europe based on the P-values?

1. How does the "missing" categorical variable (the one removed due to the rule) become interpreted once you drop a few of the ones that are included?

Remember that this is for a logistic regression but I am okay with knowing how it would work for linear as well. And would another method be more suited for this?

Thank you!