12
$\begingroup$

I was under the belief that scaling of features should not affect the result of logistic regression. However, in the example below, when I scale the second feature by uncommenting the commented line, the AUC changes substantially (from 0.970 to 0.520):

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn import metrics

cancer = load_breast_cancer()
X = cancer.data[:,0:2] # Only use two of the features
#X[:,1] = X[:,1]*10000 # Scaling
y = cancer.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)

log_reg = LogisticRegression()
log_reg.fit(X_train, y_train)

fpr, tpr, _ = metrics.roc_curve(y_test, log_reg.predict_proba(X_test)[:,1])
auc = metrics.auc(fpr, tpr)
auc

I believe this has to do with regularization (which is a topic I haven't studied in detail). If so, is there a best practice to normalize the features when doing logistic regression with regularization? Also, is there a way to turn off regularization when doing logistic regression in scikit-learn

Improve this question
$\endgroup$

1 Answer 1

Reset to default
17
$\begingroup$

I was under the belief that scaling of features should not affect the result of logistic regression. However, in the example below, when I scale the second feature by uncommenting the commented line, the AUC changes substantially (from 0.970 to 0.520) ... I believe this has to do with regularization

That is a good guess. If you look at the documentation for sklearn.linear_model.LogisticRegression, you can see the first parameter is:

penalty : str, ‘l1’ or ‘l2’, default: ‘l2’ - Used to specify the norm used in the penalization. The ‘newton-cg’, ‘sag’ and ‘lbfgs’ solvers support only l2 penalties.

Regularization makes the predictor dependent on the scale of the features.

If so, is there a best practice to normalize the features when doing logistic regression with regularization?

Yes. The authors of Elements of Statistical Learning recommend doing so. In sklearn, use sklearn.preprocessing.StandardScaler.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Perfect! from sklearn.preprocessing import StandardScaler; X_train, X_test, y_train, y_test = train_test_split(StandardScaler().fit_transform(X), y, test_size=0.33, random_state=42) gives an AUC of 0.972, greater than in both of the earlier cases. $\endgroup$
    – matthiash
    Commented Jul 11, 2017 at 14:59
  • 3
    $\begingroup$ @matthiash Great! However, I strongly encourage you to read up on sklearn pipelines. They will allow you to do this right also on cross validations on stuff. Pipelines are a cornerstone of sklearn, IMHO. $\endgroup$
    – Ami Tavory
    Commented Jul 11, 2017 at 15:02
  • 2
    $\begingroup$ @AmiTavory can you please specify where the authors of ESL states that features normalization is recommended? $\endgroup$
    – se7entyse7en
    Commented Aug 9, 2017 at 15:16
  • 4
    $\begingroup$ @se7entyse7en In the 2nd edition, it is in Secion 3.4 (Shrinkage Methods), on the penultimate paragraph of page 63: "The ridge solutions are not equivariant under scaling of the inputs, and so one normally standardizes the inputs before solving...". $\endgroup$
    – Ami Tavory
    Commented Aug 11, 2017 at 8:41
  • $\begingroup$ Are there any arguments in favour of generally using standard scaling over min-max scaling? $\endgroup$
    – zr0gravity7
    Commented Oct 25, 2021 at 18:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.