2
$\begingroup$

Question of the day: I'm running a logistic regression (results below), and I come across a coefficient that is insanely large (in absolute terms). Usually, we don't care about things like that because it's simply a question of scale - if it's really bothering you - multiply this column by 1000 and the coefficient will become -2.7. But if it's not bothering you - leave it.

However, today I needed to investigate the impact of the IV's on the odds ratio. As usual, you take the exponent of each coefficient and that will be the multiplying effect on your odds in the case of a 1-unit change in the underlying IV. In this case, since the coefficient is so negative exp(coeff) = 0

Questions:

  1. Does this mean that we can no longer not care about scale and need to always aim for the coefficients to be somewhere between -5 and 5 tops? (if we need to be able to interpret effect on the odds-ratio)

  2. [The important one] Say, I want to give somebody simplistic advice on how to behave in this situation. The obvious answer is: fix the scale of that specific column. But is there a more general approach? Is there something they can do to never end up in this situation? I'll specify the question: what is the best step-by-step way of interpreting coefficients (in terms of impact on odds-ratio) that guarantees you won't have problems with exp(beta) being 0 or too high?

  3. The IV in question, spendratio2, is the ratio of monthly credit card expenditure to yearly income. It ranges from 0.0001 to 0.9 with a mean of 0.068 and stddev of 0.094. Why I bring this up is that it's not kilometres or kilograms, and multiplying a ratio by 1000 might make it hard to understand and explain to business users. What are your suggestions for changing scale of ratios?

Thank you,

Kind regards,

Kirill Eremenko

Logistic Regression Coeff Too High

Improve this question
$\endgroup$
1

3 Answers 3

Reset to default
4
$\begingroup$

One likely explanation which has not been offered is that you have quasi-separation. For nearly all the values of spendratio2 above a threshold cardhldr has the value 1 and for nearly all the values below a threshold, not necessarily the same it has the value 0 (or vice versa). Plotting the proportion of cardhldr for a number of categories of spendratio2 should clarify whether I am right.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ I'd also suggest a full separation, which more likely to appear for very sparse features. Plotting the proportion should help make it clear $\endgroup$
    – Oleg
    Commented Jan 15, 2020 at 14:15
4
$\begingroup$

With an SD of 0.09, a one-unit change is too large. I suggest using the inter-quartile-range of the predictor.

But for ratios you should almost always take logs, and even that is unlikely to behave linearly. I recommend using a regression spline of the log ratio. Inter-quartile-range odds ratios can readily be computed even with nonlinear effects.

Improve this answer
$\endgroup$
0
$\begingroup$

The exponent is only very close to zero, not exactly zero. I don't see any problem with interpreting the odds ratio sensitivity the way you did it. Rescaling the feature by 1/c is just asking for the impact of c units of the IV.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Hi, thanks for the quick response. What do you mean by "rescaling the feature by 1/c is just asking for the impact of c units of the IV"? $\endgroup$
    – Kirill Eremenko
    Commented Aug 22, 2015 at 5:51
  • $\begingroup$ $exp(coeff) \rightarrow 0$ means your per unit change of the IV have huge effect on your outcome variable. It is has the same effect as $exp(coeff) \rightarrow \infty$ if only you change your reference group. The relation is $\frac{1}{\infty}=0$ Compare coefficient of IV directly just like compare apples to oranges. $\endgroup$
    – Deep North
    Commented Aug 22, 2015 at 7:34
  • $\begingroup$ Ok. Any comments on part 2 of my question? $\endgroup$
    – Kirill Eremenko
    Commented Aug 22, 2015 at 7:55
  • $\begingroup$ Rescaling the value of a feature is the same as rescaling the learned regression coefficient inversely, no? I am often surprised how many people don't understand this. It's like saying if your feature unit is meter, then your regressor unit is (meter)^{-1}. So in theory there is no need to rescale, unless your particular logistic solver uses the scale of the feature to speed things up. $\endgroup$
    – user48225
    Commented Aug 22, 2015 at 15:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.