How do I handle predictor variables from different distributions in logistic regression?

Question

I am using logistic regression to predict y given x1 and x2:

z = B0 + B1 * x1 + B2 * x2
y = e^z / (e^z + 1)

How is logistic regression supposed to handle cases in which my variables have very different scales? Do people ever build logistic regression models with higher-order coefficients for variables? I'm imagining something like this (for two variables):

z = B0 + B1 * x1 + B2 * x1^2 + B3 * x2 + B4 * x2^2

Alternatively, is the right answer to simply normalize, standardize or rescale the x1 and x2 values before using logistic regression?

Answer 1

Of course you can normalize your parameters, this would also increase the speed of the learning algorithm.

In order to have comparable $\beta$ at the end of the execution of the algorithm you should, for each feature $x_i$, compute its mean $\mu_i$ and its range $r_i = \max_i - \min_i$. Then you change each $r[x_i]$ value, ie the value of feature $x_i$ for a record $r$, with: $$\frac{r[x_i] - \mu_i}{r_i}$$ Now your $r[x_i]$ values lie in the interval [-1,1], so you can compare your $\beta$ with more confidence and thus your odds ratio. This also shorten the time to find the best set of $\beta$ if you are using gradient descent. Just remember to normalize your features if you want to predict the class of a new record $r'$.

You can also add higher order features but this lead to overfitting. Usually, as long as you add more parameters is better to add regularization, that try to avoid overfitting by decreasing the magnitude of your $\beta$. This is obtained adding this term to the logistic regression cost function $$\lambda\sum_{i=0}^{n}\beta_i^2$$ where $\lambda$ tune the power of the regularization.

I would suggest to have a look to Stanford's classes about machine learning here: http://www.ml-class.org/course/video/preview_list, Unit 6 and 7.

Answer 2

@Simone makes some good points, so I will just throw in a couple of complementary tidbits. Although normalization can help with things like speed, logistic regression does not make assumptions about the distributions of your predictor variables. Thus, you don't have to normalize. Second, while adding a squared term can lead to overfitting (and you need to be cautious about that) it is permissible. What that would mean is that the probability of success is higher in the middle of a predictor's range than at the extremes (or vice versa).

Answer 3

In theory, the scale of your inputs are irrelevant to logistic regression. You can "theoretically" multiply $X_1$ by $10^{10^{10^{10}}}$ and the estimate for $\beta_1$ will adjust accordingly. It will be $10^{-10^{10^{10}}}$ times smaller than the original $\beta_1$, due to the invariance property of MLEs.

But try getting R to do the above adjusted regression - it will freak out (won't even be able to construct the X matrix).

This is a bit like the cholesky decomposition algorithm for calculating a matrix square root. Yes, in exact mathematics, cholesky decomposition never involves taking square root of negative number, but round off errors, and floating point arithmetic may lead to such cases.

You can take any linear combination of your X variables, and the predicted values will be the same.

If we take @simone's advice, and using the re-scaled X variables for fitting the model. But we can use the invariance property of MLE to get the beta that we want, after using numerically stable input X variables. It may be that the beta on the original scale may be easier to interpret than the beta on @simone's transformed one. So, we have the transformed $x_{ij}$ ($i$th observation for the $j$th variable), call it $\tilde{x}_{ij}$, defined by:

$$\tilde{x}_{ij}=a_{j}x_{ij}+b_{j}$$

@simone's choice corresponds to $a_{j}=\frac{1}{x_{[N]j}-x_{[1]j}}$ and $b_j=\frac{\overline{x}_{j}}{x_{[N]j}-x_{[1]j}}$ (using $x_{[i]j}$ to denote the $i$th order statistic of the $j$th variable, i.e. $x_{[N]j}\geq x_{[N-1]j}\geq\dots\geq x_{[1]j}$). The $a_j$ and $b_j$ can be thought of as algorithm parameters (chosen to make the algorithm more stable and/or run faster). We then fit a logistic regression using $\tilde{x}_{ij}$, and get parameter estimates $\tilde{\beta}_j$. Thus we write out the linear predictor:

$$z_i = \tilde{\beta}_0 + \sum_j\tilde{x}_{ij}\tilde{\beta}_j$$

Now substitute the equation for $\tilde{x}_{ij}$ and you get:

$$z_i = \tilde{\beta}_0 + \sum_j(a_{j}x_{ij}+b_{j})\tilde{\beta}_j=\beta_0+\sum_jx_{ij}\beta_j$$ Where $$\begin{array}{c c}\beta_0=\tilde{\beta}_0+\sum_jb_{j}\tilde{\beta}_j & \;\;\;\;\;\; & \beta_j=a_j\tilde{\beta}_j \end{array}$$

You can see that theoretically, the parameters $a_j,b_j$ make no difference at all: any choice (apart from $a_j=0$) will lead to the same likelihood, because the linear predictor is unchanged. It even works for more complicated linear transforms, such as representing the X matrix by its principal components (which involves rotations). So we can back-transform the results to get the betas that we want for interpretation.