5
$\begingroup$

I was under the belief that scaling of features should not affect the result of logistic regression. However, in the example below, when I scale the second feature by uncommenting the commented line, the AUC changes substantially (from 0.970 to 0.520):

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn import metrics

cancer = load_breast_cancer()
X = cancer.data[:,0:2] # Only use two of the features
#X[:,1] = X[:,1]*10000 # Scaling
y = cancer.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)

log_reg = LogisticRegression()
log_reg.fit(X_train, y_train)

fpr, tpr, _ = metrics.roc_curve(y_test, log_reg.predict_proba(X_test)[:,1])
auc = metrics.auc(fpr, tpr)
auc

I believe this has to do with regularization (which is a topic I haven't studied in detail). If so, is there a best practice to normalize the features when doing logistic regression with regularization? Also, is there a way to turn off regularization when doing logistic regression in scikit-learn

$\endgroup$
9
$\begingroup$

I was under the belief that scaling of features should not affect the result of logistic regression. However, in the example below, when I scale the second feature by uncommenting the commented line, the AUC changes substantially (from 0.970 to 0.520) ... I believe this has to do with regularization

That is a good guess. If you look at the documentation for sklearn.linear_model.LogisticRegression, you can see the first parameter is:

penalty : str, ‘l1’ or ‘l2’, default: ‘l2’ - Used to specify the norm used in the penalization. The ‘newton-cg’, ‘sag’ and ‘lbfgs’ solvers support only l2 penalties.

Regularization makes the predictor dependent on the scale of the features.

If so, is there a best practice to normalize the features when doing logistic regression with regularization?

Yes. The authors of Elements of Statistical Learning recommend doing so. In sklearn, use sklearn.preprocessing.StandardScaler.

$\endgroup$
  • $\begingroup$ Perfect! from sklearn.preprocessing import StandardScaler; X_train, X_test, y_train, y_test = train_test_split(StandardScaler().fit_transform(X), y, test_size=0.33, random_state=42) gives an AUC of 0.972, greater than in both of the earlier cases. $\endgroup$ – matthiash Jul 11 '17 at 14:59
  • 1
    $\begingroup$ @matthiash Great! However, I strongly encourage you to read up on sklearn pipelines. They will allow you to do this right also on cross validations on stuff. Pipelines are a cornerstone of sklearn, IMHO. $\endgroup$ – Ami Tavory Jul 11 '17 at 15:02
  • 2
    $\begingroup$ @AmiTavory can you please specify where the authors of ESL states that features normalization is recommended? $\endgroup$ – se7entyse7en Aug 9 '17 at 15:16
  • 3
    $\begingroup$ @se7entyse7en In the 2nd edition, it is in Secion 3.4 (Shrinkage Methods), on the penultimate paragraph of page 63: "The ridge solutions are not equivariant under scaling of the inputs, and so one normally standardizes the inputs before solving...". $\endgroup$ – Ami Tavory Aug 11 '17 at 8:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.