Logistic regression and scaling of features

I was under the belief that scaling of features should not affect the result of logistic regression. However, in the example below, when I scale the second feature by uncommenting the commented line, the AUC changes substantially (from 0.970 to 0.520):

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn import metrics

X = cancer.data[:,0:2] # Only use two of the features
#X[:,1] = X[:,1]*10000 # Scaling
y = cancer.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)

log_reg = LogisticRegression()
log_reg.fit(X_train, y_train)

fpr, tpr, _ = metrics.roc_curve(y_test, log_reg.predict_proba(X_test)[:,1])
auc = metrics.auc(fpr, tpr)
auc


I believe this has to do with regularization (which is a topic I haven't studied in detail). If so, is there a best practice to normalize the features when doing logistic regression with regularization? Also, is there a way to turn off regularization when doing logistic regression in scikit-learn

I was under the belief that scaling of features should not affect the result of logistic regression. However, in the example below, when I scale the second feature by uncommenting the commented line, the AUC changes substantially (from 0.970 to 0.520) ... I believe this has to do with regularization

That is a good guess. If you look at the documentation for sklearn.linear_model.LogisticRegression, you can see the first parameter is:

penalty : str, ‘l1’ or ‘l2’, default: ‘l2’ - Used to specify the norm used in the penalization. The ‘newton-cg’, ‘sag’ and ‘lbfgs’ solvers support only l2 penalties.

Regularization makes the predictor dependent on the scale of the features.

If so, is there a best practice to normalize the features when doing logistic regression with regularization?

Yes. The authors of Elements of Statistical Learning recommend doing so. In sklearn, use sklearn.preprocessing.StandardScaler.

• Perfect! from sklearn.preprocessing import StandardScaler; X_train, X_test, y_train, y_test = train_test_split(StandardScaler().fit_transform(X), y, test_size=0.33, random_state=42) gives an AUC of 0.972, greater than in both of the earlier cases. Jul 11 '17 at 14:59
• @matthiash Great! However, I strongly encourage you to read up on sklearn pipelines. They will allow you to do this right also on cross validations on stuff. Pipelines are a cornerstone of sklearn, IMHO. Jul 11 '17 at 15:02
• @AmiTavory can you please specify where the authors of ESL states that features normalization is recommended? Aug 9 '17 at 15:16
• @se7entyse7en In the 2nd edition, it is in Secion 3.4 (Shrinkage Methods), on the penultimate paragraph of page 63: "The ridge solutions are not equivariant under scaling of the inputs, and so one normally standardizes the inputs before solving...". Aug 11 '17 at 8:41