78
$\begingroup$

My question is do we need to standardize the data set to make sure all variables have the same scale, between [0,1], before fitting logistic regression. The formula is:

$$\frac{x_i-\min(x_i)}{\max(x_i)-\min(x_i)}$$

My data set has 2 variables, they describe the same thing for two channels, but the volume is different. Say it's the number of customer visits in two stores, y here is whether a customer purchases. Because a customer can visit both stores, or twice first store, once second store before he makes a purchase. but the total number of customer visits for 1st store is 10 times larger than the second store. When I fit this logistic regression, without standardization, coef(store1)=37, coef(store2)=13; if I standardize the data, then coef(store1)=133, coef(store2)=11. Something like this. Which approach makes more sense?

What if I am fitting a decision tree model? I know tree structure models don't need standardization since the model itself will adjust it somehow. But checking with all of you.

Improve this question
$\endgroup$
11
  • 14
    $\begingroup$ You don't need to standardize unless your regression is regularized. However, it sometimes helps interpretability, and rarely hurts. $\endgroup$
    – alex
    Commented Jan 23, 2013 at 17:02
  • 5
    $\begingroup$ Isn't the usual way to standardize $\frac{x_i-\bar{x}}{sd(x)}$? $\endgroup$
    – Peter Flom
    Commented Jan 23, 2013 at 17:28
  • 2
    $\begingroup$ @Peter, that's what I thought before, but I found an article benetzkorn.com/2011/11/data-normalization-and-standardization/…>, it seems that normalization and standardization are different things. One is to make mean 0 variance 1, the other is to rescale each variable. That's where I get confused. Thanks for your reply. $\endgroup$
    – user1946504
    Commented Jan 23, 2013 at 17:56
  • 12
    $\begingroup$ To me standardization makes interpretation much more difficult. $\endgroup$
    – Frank Harrell
    Commented Jan 23, 2013 at 20:49
  • 2
    $\begingroup$ To clarify on what @alex said, scaling your data means the optimal regularisation factor C changes. So you need to choose C after standardising the data. $\endgroup$
    – akxlr
    Commented Aug 21, 2015 at 14:07

4 Answers 4

Reset to default
63
$\begingroup$

Standardization isn't required for logistic regression. The main goal of standardizing features is to help convergence of the technique used for optimization. For example, if you use Newton-Raphson to maximize the likelihood, standardizing the features makes the convergence faster. Otherwise, you can run your logistic regression without any standardization treatment on the features.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for your reply. Does that mean standardization is preferred? Since we definitely want the model converge and when we have millions of variables, it's just easier to implement the logic of standardization in the modeling pipeline than tuning the variables one by one as needed. Am I understanding right? $\endgroup$
    – user1946504
    Commented Sep 18, 2014 at 21:33
  • 5
    $\begingroup$ that depends on the purpose of the analysis. Modern software can handle pretty extreme data without standardizing. If there is a natural unit for each variables (years, euros, kg, etc.) then I would be hesitant to standardize, though I feel free to change the unit from kg to for example tons or grams whenever that makes more sense. $\endgroup$
    – Maarten Buis
    Commented Nov 18, 2014 at 8:43
  • $\begingroup$ If using scikit-learn, you should think about standardizing, because sklearn.linear_model.LogisticRegression uses L2-penalty by default, which is Ridge Regression. Here, it makes a difference whether you standardize, according to other answers. $\endgroup$
    – Benji
    Commented Jul 19, 2022 at 9:36
43
$\begingroup$

If you use logistic regression with LASSO or ridge regression (as Weka Logistic class does) you should. As Hastie,Tibshirani and Friedman points out (page 82 of the pdf or at page 63 of the book):

The ridge solutions are not equivariant under scaling of the inputs, and so one normally standardizes the inputs before solving.

Also this thread does.

Improve this answer
$\endgroup$
27
$\begingroup$

@Aymen is right, you don't need to normalize your data for logistic regression. (For more general information, it may help to read through this CV thread: When should you center your data & when should you standardize?; you might also note that your transformation is more commonly called 'normalizing', see: How to verify a distribution is normalized?) Let me address some other points in the question.

It is worth noting here that in logistic regression your coefficients indicate the effect of a one-unit change in your predictor variable on the log odds of 'success'. The effect of transforming a variable (such as by standardizing or normalizing) is to change what we are calling a 'unit' in the context of our model. Your raw $x$ data varied across some number of units in the original metric. After you normalized, your data ranged from $0$ to $1$. That is, a change of one unit now means going from the lowest valued observation to the highest valued observation. The amount of increase in the log odds of success has not changed. From these facts, I suspect that your first variable (store1) spanned $133/37\approx 3.6$ original units, and your second variable (store2) spanned only $11/13\approx 0.85$ original units.

Improve this answer
$\endgroup$
1
$\begingroup$

In some cases, you have to normalize/standardize the input data, especially if there are different large scales between the features. A use case I faced recently, trying to fit a logistic regression to the data in the figure below:

Dataset to be fit to the Logistic Regression

Here are the values of the intercept, w1 for the Age and w2 for EstimatedSalary, (array([-2.24944689e-10]), array([[-2.10415172e-09, -2.69301403e-06]])), The model has dismissed/neglected the age feature, Decision boundary in this case is illustrated in Figure below

Decision boundary no normalization

Now, let's apply MinMax normalization on the Age and EstimatedSalary columns, Here are the intercept, w1, and w2 values (array([-4.55319723]), array([[5.535191 , 2.77997059]])) and below the new decision boundary

enter image description here

That's all, I hope this can help you guys to understand that sometimes scaling (normalization / standardization) is very important! Thank you

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.