Should you scale the dataset (normalization or standardization) for a simple multiple logistic regression model?

Question

I have read a lot of conflicting literature about scaling the dataset (using methods such as normalization or standardization) for a multiple logistic regression model, and I am wondering if scaling is necessary for this type of machine learning model. For example, this source (Is it a good practice to always scale/normalize data for machine learning?) supports the use of scaling methods for logistic regression:

For machine learning models that include coefficients (e.g. regression, logistic regression, etc) the main reason to normalize is numerical stability. Mathematically, if one of your predictor columns is multiplied by 10^6, then the corresponding regression coefficient will get multiplied by 10^{-6} and the results will be the same.

This source (Is normalizing the features always good for classification?) states that parametric algorithms such as logistic regression should implement scaling methods:

Typically decision trees (for instance C4.5, implemented as J48 in Weka you used) are non parametric, that is they don't make any assumption regarding the distribution of the data. As long as the normalization doesn't change the ranks of the data (and I know of no normalization that does that), the results will be exactly the same (you will only get different splitting levels). Of course this doesn't hold for algorithms making parametric assumptions (logistic regression, etc.) So you shouldn't always normalize, but you should decide to do it or not depending on your algorithm.

And this source (When conducting multiple regression, when should you center your predictor variables & when should you standardize them?) has an answer that also supports the use of scaling:

In regression, it is often recommended to center the variables so that the predictors have mean 0. This makes it easier to interpret the intercept term as the expected value of Yi when the predictor values are set to their means.

However, in the same source (When conducting multiple regression, when should you center your predictor variables & when should you standardize them?), another answer states that scaling is not necessary:

You have come across a common belief. However, in general, you do not need to center or standardize your data for multiple regression. Different explanatory variables are almost always on different scales (i.e., measured in different units). This is not a problem; the betas are estimated such that they convert the units of each explanatory variable into the units of the response variable appropriately.

Another source (What algorithms need feature scaling, beside from SVM?) referenced a list from dataschool.io that states that logistic regression does not need feature scaling unless it is regularized. The list from dataschool.io can be found here: https://www.dataschool.io/comparing-supervised-learning-algorithms/

Could someone clarify whether or not I should use scaling methods on the dataset for a multiple logistic regression model? Thank you so much for taking the time to read this.

Answer 1

The following summarizes the multiple references you provide, with respect to your case of "simple" (unpenalized) multiple logistic regression.

For multiple logistic regression or other unpenalized regression models, there is no general need to pre-scale. For regression models there is an intelligible interpretation of the coefficients in terms of the original scales of the predictors. Any pre-scaling removes that intelligibility unless you back-transform the coefficients to represent the predictors in their original scales.

Numerically large or small values of predictors can lead to problems with numerical stability, particularly when calculations involve exponentiation. In that case you might need to standardize first, but afterward re-express coefficients back in the original scales. Some implementations might do that "under the hood" to avoid problems, like the coxph() function for survival analysis in R.

Other approaches might need (some) pre-scaling, as explained in the references you cite.

In the context of regression, penalized approaches like ridge regression or LASSO are intended to evaluate all predictors on comparable scales. The default of some implementations is thus to pre-standardize while choosing the penalty factor, and then rescale when reporting coefficients.

But there's no need to accept that default in penalized models. You can choose individual scaling factors for predictors instead, as shown on this page. If binary or categorical predictors are involved in a penalized model, you have to think carefully about just what you mean by having predictors on comparable scales, as discussed here, and recognize that, for multi-level categorical predictors, usual standardization methods might give different results based on the choice of reference category. You also can choose not to penalize predictors of primary interest, while restricting penalization (and implicit scaling/re-scaling) to predictors for which you simply want to control.

Some non-regression modeling methods might need predictors to be pre-adjusted to comparable scales, but that's not the case for your "simple" multiple logistic regression, except perhaps to overcome numerical stability problems.