Why using StandardScaler from sklearn before LogisticRegression increase avg cross_val_score? Standarization should only help in faster convergece

Question

I am using UC Irvine ML Glass Identification dataset mentioned in a book "Applied Predictive Modelling".

I tried rudimentary logistic regression models using sklearn with and without the StandardScaler class and had 3% improvement in accuracy when cv =5.

I did use pipeline to not leak the data but i am not sure if i leaked it somehow.

lr = LogisticRegression(max_iter=10000, random_state=42)
results = cross_val_score(estimator=lr,cv=5,scoring="accuracy", X = glass_data.iloc[:,0:-1], y=glass_data.iloc[:,-1] )
# had to increase max iterations as data is unprocessed, gradient descent is too slow because of it.
np.mean(results)

above code had lower accuracy of .579

from sklearn.preprocessing import StandardScaler
 # since we did not divide our data set and are using cross validation, we need to use a pipeline 
# otherwise there will be data leakage 
from sklearn.pipeline import Pipeline

clf = Pipeline([("scaler", StandardScaler()), ("lr", LogisticRegression( random_state=42))])
result = cross_val_score(clf, cv=5, scoring="accuracy", X=glass_data.iloc[:,0:-1], y=glass_data.iloc[:,-1])
np.mean(result)

this had accuracy of .6078

Also, i did try to increase max_iter to a very large number for case 1 but the cross_val_score average was always the same.

Answer 1

By default, sklearn logistic regression is penalized. From the documentation:

penalty{‘l1’, ‘l2’, ‘elasticnet’, ‘none’}, default=’l2’

Specify the norm of the penalty:

• 'none': no penalty is added;
• 'l2': add a L2 penalty term and it is the default choice;
• 'l1': add a L1 penalty term;
• 'elasticnet': both L1 and L2 penalty terms are added.

The penalty is applied to the coefficients. Changing the scale of the data changes the coefficients, which implies a different penalty, which implies a different fit.

To fit a model without a penalty, use penalty='none'. The maximum likelihood fit of the model will be the same regardless of (nonzero) scaling applied to the features.

The sharp corner here is computational. Poorly-conditioned models may take many iterations to attain the maximum likelihood (OP has already discovered this!). If one model doesn't attain the maximum, then we shouldn't be surprised if it doesn't agree with a model that does.