5
$\begingroup$

I am estimating a binary logistic regression with L1 norm. According to the regression coefficients, the sign of x1's coefficient is positive. However, the descriptive statistics indicates that the mean of x1 variable for class 0 is greater than that of class 1.

Could you please explain me how to justify this contradictory results?

It should be noted that descriptive statistics are calculated before the standardization of independent variables.

$\endgroup$
  • $\begingroup$ Does your logistic regression contain more than one feature? $\endgroup$ – Matthew Drury Aug 20 '18 at 6:09
  • $\begingroup$ @MatthewDrury Thanks. Yes, it contains a lot of variables, so I used L1 norm (Lasso) to select some important ones. $\endgroup$ – ebrahimi Aug 20 '18 at 6:12
7
$\begingroup$

This not an issue with L1 norms nor with logistic regression specifically; it will happen with ordinary L2-norm multiple linear regression. The direction of an unconditional relationship can change once you condition on another variable (include an additional feature).

Consider 4 groups, where the $p=\text{Pr}(Y=1)$ is an increasing function of $x$ within each group but where the four groups get lower average $p$ as x increases:

Plot of P(Y=1) vs x for four groups; the group means get lower as x increases but within each group the relationship is positive

Of course the population $p$'s are not observable in practice; you only observe the proportion of Y's that are 1. However, if you have many replicates at each x-value (say 50 or something), you could actually observe something very like this for the sample proportions at each x..

Now if you fit a logistic to just the y vs x values, ignoring the group factor, you will get a negative coefficient. But as soon as you include group, the coefficient for x will be positive.

[In fact it is possible that the sign could even flip each time you included additional feature.]

Further reading:

https://en.wikipedia.org/wiki/Simpson%27s_paradox

https://en.wikipedia.org/wiki/Omitted-variable_bias

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Nice example Glen. $\endgroup$ – Matthew Drury Aug 20 '18 at 16:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.