feature selection of sub-categorical data on a linear regression model

Question

I have a data-set with 7 features, 6 numeric and 1 categorical. the categorical data is "Species", which has the ability to be sub-categorized (species, genus, family, order, etc...).

I want to build a linear regression model from this dataset. How should I decide which dummy variables to include in the final model? Can I mix dummy variables from different category levels? If I include a dummy variable from one category level like (species=dog) do I need to include all dummy variables from that level (species=cat), or can I just roll up the non-significant ones into "Other"?

My initial thought was to start with ALL the variables numeric and dummy from all the possible levels (there will be ~100 dummy variables from all levels), and all treated equally. Then build the model using variable addition selecting the most significant variable at each step, till I got to a P value, on the new variable I felt like stopping at (likely 0.01).

After that I was considering using target shuffling on the final model and eliminating any variables that showed a P of 0.01 or more.

Are there any other specific issues I need to be aware of when dealing with sub-categories like this on a linear regression model?

Answer 1

You could replace the one categorical variable "species" and replace it with multiple variables: order, family, genus, species, ... and then use a nested variable coding: species within genus within family ... You will then get coefficients for family, say, then for genus within family, then for species within genus within family, ... Such parameter coding may give more interpretable results. To make this more concrete, this requires a nested coding scheme, as in a nested design or How do you deal with "nested" variables in a regression model?. In R, with three factor variables Family, Genus and Species this can be specified with the model formula

~  Family + Genus:Family + Species:Genus:Family

Another approach is to use the data to join levels with similar effects in the model, this can be done with the fused lasso, see Principled way of collapsing categorical variables with many levels? and Preprocess categorical variables with many values

The approach you are proposing in the question, variable selection based on significance testing, is generally frowned upon and does not give much meaning in this setting. Basically this is because the meaningful variables in modeling is the factor variable as a totality, and not its individual levels. If you drop some dummy/indicator variable coding for some specifil level of the factor, that is changing the meaning of the factor variable, so changing the meaning of the model. See Can I ignore coefficients for non-significant levels of factors in a linear model? for details.