Effects of standardizing variables before regularized logistic regression on results?

Question

I am doing regularized logistic regression (elastic net) on a set of variables that have different ranges (i.e. one ranges from 0~1, another ranges from 2~6, another ranges in the hundreds, and some involves negative numbers as well).

I understand that in a case like this where each variable has very different range of values, you should standardize the variables before the regression. Thus I have done so using Rescaling, where rescaled x = ( x - min(x))/( max(x)-min(x)).

Before I standardized my variables, the results of my logistic regression (nested 10-fold CV), measured as area under the curve, was in the 0.80 range. However, after I standardized my variables, it has dropped to just chance level (0.50 range).

Is it possible for AUC to plummet this much after variable standardization? Or is it more likely that I made a mistake in editing my code? (I edited a few lines in cvglmnet.m and cvlognet.m in the glmnet package for Matlab).

**On a side note, I am standardizing the variables so that they are rescaled to have values of 0~1. The min and max values are found for each feature in the training set, and based on this min and max value I rescaled the values of both the training set and test set. However, I noticed that when I do this, I get values that are slightly beyond the 0~1 range in the test set. I am wondering is this correct?

Answer 1

Lets assume you have $9 \over 10 $ th of the dataset set aside as the training dataset $a^{train}$ and $1 \over 10 $ th as the test dataset $a^{test}$ for a one variable regression model.

If the minimum value of this variable is $min_{train}$ and maximum value is $max_{train}$ , then the training set should be scaled as follows:

$a_{scaled}^{train} = {a^{train} - min_{train} \over max_{train}- min_{train}} $

After, you've found the best fit for this training dataset, you should scale the test set with the exact same scaling parameters before using it to test its accuracy. So the test set will also be similarly scaled:

$a_{scaled}^{test} = {a^{test} - min_{train} \over max_{train}- min_{train}} $

Its normal to see some values beyond the 0~1 range in the test set after applying the scaling; this may happen due to extreme values in the test set beyond the min/max values of the training set.

You seem to have used the correct logic, but I suspect you may have jumbled the test dataset somehow. For example, you might be comparing true value $y^i$ against estimate $\hat{y}^j$ , which could explain AUC value dropping to a random chance range near 0.5 .