4
$\begingroup$

With my limited understanding of the logistic regression, I understand that the coefficients in logistic regression are the odds ratios. Does it make send to normalize them (divide each one over the overall sum)? I guess NO, can anyone explain why? in this case is there a way to relate the different predictors' coefficients to have a comparative understanding?

$\endgroup$
  • $\begingroup$ Instead of asking for justification for something that doesn't seem to me to make sense please provide justification or motivation for such normalization. The short answer to your question is no, it makes no sense to do that. $\endgroup$ – Frank Harrell Jun 18 '16 at 11:30
  • $\begingroup$ in the case of binary logistic regression, my purpose is to measure the relative "contribution" of each predictor(independant veribale) as a descriptive feature in the logistic model. For instance in the case of threee predictors, I would say that predictor pred1, pred2, and pred3 contribute 60%,30%, and 10% to the model. is there a way to have this information in the logistic regression? I thought why I couldnt do this by normalizing the coefficients? $\endgroup$ – Abdu Jun 18 '16 at 11:44
6
$\begingroup$

The short answer is that normalizing the coefficients will not affect the predictions, but it will mess up the estimated class probabilities. Don't do it.

The coefficients don't represent the odds ratios but rather the feature weights. They can be negative. If a coefficient is strongly positive, it means that the corresponding feature is very much correlated with the positive class. If it is strongly negative, then its means that the feature is strongly correlated with the negative class. If the coefficient is close to zero, then it means that the feature is not correlated much with either the positive or the negative class.

So if you want to compare the importance of each feature, you should compare the absolute values of the coefficients (and you can normalize them just for convenience, if you want, but don't use these normalized absolute coefficients to make predictions, only use them to compare feature importance). (Edit: this assumes that the features have been normalized prior to training)

This is probably all you need to know. Read on if you want to understand what would happen if you tried to normalize the coefficients.

The decision function for logistic regression is:

$h_\mathbf{\theta}(\mathbf{x}) = \sigma(\sum\limits_{i=0}^{n}\theta_i x_i)$

where $\sigma(t) = \dfrac{1}{1 + \exp(-t)}$ (the logistic function)

and $\mathbf{\theta}$ is the parameter vector, and $\mathbf{x}$ is the feature vector (including a bias term $x_0 = 1$) and $n$ is the number of features.

The model's prediction $\hat{y}$ for the instance $\mathbf{x}$ is given by:

$ \hat{y} = \begin{cases} 0 & \text{ if }h_\mathbf{\theta}(\mathbf{x}) < 0.5\\ 1 & \text{ if }h_\mathbf{\theta}(\mathbf{x}) \ge 0.5 \end{cases} $

Notice that $\sigma(t) \ge 0.5$ when $t \ge 0$, and $\sigma(t) < 0.5$ when $t < 0$ so the prediction simplifies to:

$ \hat{y} = \begin{cases} 0 & \text{ if }\sum\limits_{i=0}^{n}\theta_i x_i < 0\\ 1 & \text{ if }\sum\limits_{i=0}^{n}\theta_i x_i \ge 0 \end{cases} $

If you normalize the feature vector, you get the new parameter vector $\bar{\mathbf{\theta}} = \dfrac{\mathbf{\theta}}{K} $. Since the coefficients can be negative, it would not make sense to divide them by the sum of coefficients (the sum could be negative or zero). So instead, let's define $K$ as the range of values (anyway, even if you choose another method for normalization, it does not change what follows).

$K = \underset{i}\max(\theta_i) - \underset{i}\min(\theta_i)$

Look at what happens to the sum used for predictions:

$ \sum\limits_{i=0}^{n}\bar{\theta}_i x_i = \sum\limits_{i=0}^{n}\dfrac{\theta_i}{K} x_i = \dfrac{1}{K}\sum\limits_{i=0}^{n}\theta_i x_i $

Everything just got multiplied by the constant $\dfrac{1}{K}$.

If $K > 0$, the predictions don't change a bit, since $\dfrac{1}{K} \sum\limits_{i=0}^{n}\theta_i x_i$ has the same sign as $\sum\limits_{i=0}^{n}\theta_i x_i$. If $K = 0$ (which can only happen if all coefficients are equal), then $K$ is not defined (you can't normalize the coefficients). If you use another normalization technique, and end up with $K<0$, then all predictions get reversed, which is probably not a good idea!

So normalizing the coefficients (by dividing them by their range of values) will not affect predictions. However, it will mess up the decision function $h_\mathbf{\theta}$. This function is used to estimate the probability of the positive class. By multiplying all coefficients by $\dfrac{1}{K}$, you will end up making the same predictions (since the sign is not affected), but the estimated probability will be higher or lower depending on the value of $K$. For example, if $K = 2$ then:

$h_\mathbf{\bar{\theta}}(\mathbf{x}) = \sigma(\sum\limits_{i=0}^{n}\bar{\theta}_i x_i) = \sigma\left(\dfrac{1}{2} \sum\limits_{i=0}^{n}\theta_i x_i\right)$

This can't be simplified much: if you plot the curve of $\sigma(t)$ and compare it to $\sigma(\frac{1}{2}t)$, you will find that this updated model will be much less confident about its predictions. For no good reason.

$\endgroup$
  • $\begingroup$ thanks for your reply. I understand that the normalized coefficients will mess up the model. I just wanted to compare the predictors' contribution. And currently I was doing exactly as you recommended in "So if you want to compare the importance of each feature, you should compare the absolute values of the coefficients ... to compare feature importance)." then I'm not sure if it is sound, when commenting on the logistic regression model, to say pred1 "explain"/"contribute" to the model with 60%... would you please confirm whether it is sound or not to say this? $\endgroup$ – Abdu Jun 18 '16 at 12:33
  • 1
    $\begingroup$ Suppose there are 3 features and the coefficients (ignoring the bias term) are 1, -2, and 7. I think it is ok to say that the third feature contributes to 70% of the decision, assuming of course that the training set was normalized (or else the coefficients might be small simply to compensate huge feature values, and vice versa). Also check out Frank Harrell's interesting answer. $\endgroup$ – MiniQuark Jun 18 '16 at 15:28
4
$\begingroup$

To show the relative contribution of each predictor, and to be able to handle the common case where the predictor is represented by multiple variables (indicator variables or nonlinear terms), I prefer to make a dot chart of the chunk test partial $\chi^2$ statistic for each predictor. You can also get a meaningful chart by computing the proportion of the total model $\chi^2$ is explained by each predictor and plotting that. In the R rms package these things are simple: plot(anova(fit)).

$\endgroup$
4
$\begingroup$

The correct way to compare coefficients on the same scale is to rescale the predictors themselves. A common transformation is to turn continuous predictors into Z-scores by subtracting the sample mean and dividing by the sample standard deviation. (Gelman[1] recommends two standard deviations)

This will change the values of the coefficients, because they are now measured in "standard deviations" rather than "units" of each input. But this is in fact your goal: since they coefficients are all measured on the same scale, they will all be directly comparable. This will not change the predictions because the predictors enter into logistic regression linearly.

[1]: Gelman, A. (2008), Scaling regression inputs by dividing by two standard deviations. Statistics in Medicine, 27: 2865–2873. doi:10.1002/sim.3107 http://www.stat.columbia.edu/~gelman/research/published/standardizing7.pdf

$\endgroup$
  • $\begingroup$ thx for sharing this paper I will have a look at it $\endgroup$ – Abdu Jun 20 '16 at 4:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.