Can I use a standard scaler to compare categorical and non categorical variables in regression

Question

I am wondering if it is appropriate to use a standard scaler (center and divide by std) to scale categorical and real number variables in a data set for the purpose of comparing the resulting linear regression model coefficient variables together to determine feature importance. For example, in the Kaggle insurance dataset (with target value charges) we have real numbered columns age and bmi. We have an integer column number of children. We also have categorical columns region, sex, and smoker. Is it appropriate to one hot encode region, and encode sex and smoker as 0/1 data. Then standard scale all columns (age, bmi, children, encoded sex, encoded smoker, and one hot region columns), fit a linear regression model, and compare the resulting coefficients against each other for the purposes of feature importance analysis.

I realize that it doesn't make a lot of sense to talk about how a 1 standard deviation increase in how sex, region or smoker affects insurance charges but I'm wondering if it is still appropriate to compare scaled sex, region, and smoker predictors, against the other scaled variables like bmi, age, and num children. Actually - maybe it does make a little sense to talk about how a one standard deviation increase in smoker affects charges.

Is there anything inherently wrong with standardizing sex, smoker, and region?

Answer 1

There are big problems with trying to use individual regression coefficients to evaluate feature importance, which only become magnified when you try to scale categorical predictors.

To start, how will you use the individual coefficients for a multi-level categorial predictor like region? In dummy coding, those coefficients represent differences from the reference category. Thus the coefficients used for "feature importance" will differ depending on your choice of reference. You need a way to evaluate all levels of a categorical predictor together. Similar considerations apply when a predictor is included in interactions or in non-linear terms.

In terms of scaling categorical predictors, this page discusses the problem in the related context of pre-scaling predictors for penalized regression. With a multi-level predictor like region, the scaling will differ depending on your choice of reference level!

You can avoid these problems in a regression model by using a more appropriate evaluation of feature importance. This page illustrates use of partial chi-square statistics, corrected for the corresponding degrees of freedom. That allows inclusion of all terms involving a predictor or all levels of a categorical predictor at once, in a measure related to how much variance in outcome can be associated with such predictors.

You should recognize that any estimates of feature importance can be very unstable, changing from data sample to data sample. See Section 5.4 of Frank Harrell's course notes, for example, for the instability of feature ranking among bootstrapped samples from a single data set.