I am wondering if it is appropriate to use a standard scaler (center and divide by std) to scale categorical and real number variables in a data set for the purpose of comparing the resulting linear regression model coefficient variables together to determine feature importance. For example, in the Kaggle insurance dataset (with target value charges) we have real numbered columns age and bmi. We have an integer column number of children. We also have categorical columns region, sex, and smoker. Is it appropriate to one hot encode region, and encode sex and smoker as 0/1 data. Then standard scale all columns (age, bmi, children, encoded sex, encoded smoker, and one hot region columns), fit a linear regression model, and compare the resulting coefficients against each other for the purposes of feature importance analysis.

I realize that it doesn't make a lot of sense to talk about how a 1 standard deviation increase in how sex, region or smoker affects insurance charges but I'm wondering if it is still appropriate to compare scaled sex, region, and smoker predictors, against the other scaled variables like bmi, age, and num children. Actually - maybe it does make a little sense to talk about how a one standard deviation increase in smoker affects charges.

Is there anything inherently wrong with standardizing sex, smoker, and region?


1 Answer 1


There are big problems with trying to use individual regression coefficients to evaluate feature importance, which only become magnified when you try to scale categorical predictors.

To start, how will you use the individual coefficients for a multi-level categorial predictor like region? In dummy coding, those coefficients represent differences from the reference category. Thus the coefficients used for "feature importance" will differ depending on your choice of reference. You need a way to evaluate all levels of a categorical predictor together. Similar considerations apply when a predictor is included in interactions or in non-linear terms.

In terms of scaling categorical predictors, this page discusses the problem in the related context of pre-scaling predictors for penalized regression. With a multi-level predictor like region, the scaling will differ depending on your choice of reference level!

You can avoid these problems in a regression model by using a more appropriate evaluation of feature importance. This page illustrates use of partial chi-square statistics, corrected for the corresponding degrees of freedom. That allows inclusion of all terms involving a predictor or all levels of a categorical predictor at once, in a measure related to how much variance in outcome can be associated with such predictors.

You should recognize that any estimates of feature importance can be very unstable, changing from data sample to data sample. See Section 5.4 of Frank Harrell's course notes, for example, for the instability of feature ranking among bootstrapped samples from a single data set.

  • 1
    $\begingroup$ Is it true that that a one hot encoded dummy variable represents the difference from a reference only if one col is held out (example: one category is represented by 00 in a 3 category one hot encoded transformation). $\endgroup$
    – Willard
    Oct 30, 2022 at 18:37
  • 1
    $\begingroup$ @Willard yes. But if you don't do that you almost always end up with problems. Unless you represent one level as all 0s you end up with a linearly dependent set of one-hot-encoded columns in the design matrix. With a simple model you might get away with that by removing the intercept to obtain estimated outcomes directly for each level. In more complicated models, software will either refuse to handle the linearly dependent predictors or remove its own choice of one of them, perhaps even silently. Safest to choose your own reference level and do comparisons of interest after modeling. $\endgroup$
    – EdM
    Oct 30, 2022 at 20:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.