I thought this would be an easy question to find an answer to, but for the life of me I am having trouble finding anything that fully addresses my current problem:

Consider a situation where we are trying to measure the effects that a set of actions a person can take have on a binary outcome variable. There are 7 such actions, all of which can be taken multiple times (i.e. they are each bounded [0, infinity]). This leads us to a setup along the lines of:

Y ~ b0 + b1 * Action 1 + b2 * Action 2 + b3 * Action 3 + b4 * Action 4 + b5 * Action 5 + b6 * Action 6 + b7 * Action 7 + error

The piece that is causing me (and my collaborators) grief, is that there is collinearity between the number of times these actions are taken. Someone who takes Action 1 frequently is also more likely to take the other actions more often than the average person. Further the distributions of the frequencies of these actions are not balanced. Action 1 is most prevalent, Action 2 less so, and the rest are relatively rare. For context, here is the correlation matrix:

enter image description here

In an ideal world we would like to generate some sort of weighting scheme from the coefficients generated by this setup, where we can compare the relative effects that each action has on the outcome variable. However, due to this collinearity we are uncertain if we can trust the coefficients as ‘weights’.

Poking around we have found that collinearity will increase the variance of our coefficients (pg 126), but we have a large enough sample that this high variance does not leave us with insignificant coefficients. What is unclear to us is if collinearity will not only increase the variance of our coefficients, but also lead to ‘incorrect’ weights. I.e. will Action 1’s coefficient ‘soak up’ some of the effect of Action 2 because Action 1 carries the strongest signal and the two actions are correlated?

Any thoughts on this would be much appreciated, been going back and forth on it for longer than I would like to admit…Thanks!!

  • 1
    $\begingroup$ The "some sort of weighting scheme" you seek is called the estimated coefficients in the regression. The model already accounts for all the mutual (linear) effects of the explanatory variables on each other. That's part of what we mean when we say multiple regression "controls" for the other variables. $\endgroup$
    – whuber
    Mar 31 at 21:43
  • $\begingroup$ Thanks @whuber. This was our first intuition, but I ran into trouble when trying to contextualize the correct way to think about the how these coefficients shift when we include, say, only 2 actions in the regression as opposed to 3 (or as opposed to 7). We have observed the 3rd action's coefficient shift from negative (when included by itself) to positive (when included with actions 1 and 2), which feels like we aren't solely capturing that action's effect, but it's effect after action 1 and 2 have already taken the negative part of that effect into their coefficients...am I off-base here? $\endgroup$ Mar 31 at 23:06
  • $\begingroup$ Can we really assign true individual 'weights' to each action under these circumstances? Or is the third action just correcting our prediction in aggregate, but Action 1 and Action 2 are sucking up some of Action 3's causal effect? $\endgroup$ Mar 31 at 23:07
  • $\begingroup$ You're not off base. This isn't about logistic regression: it's a fact of multiple regression generally that unless the explanatory variables are orthogonal, the coefficient of any one of them depends on all the other variables in the regression. $\endgroup$
    – whuber
    Apr 1 at 12:39

1 Answer 1


My first thought would be you came across a phenomenon called correlation bias. This means if two features are highly collinear, their weights will be underestimated in a logistic regression model learning process. I am not super familiar with this phenomenon, so I cannot answer in detail at this moment. I hope my comments poke a helpful direction for you to investigate further. Thanks!

  • $\begingroup$ Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Sep 7 at 9:58
  • $\begingroup$ This does not really answer the question. If you have a different question, you can ask it by clicking Ask Question. To get notified when this question gets new answers, you can follow this question. Once you have enough reputation, you can also add a bounty to draw more attention to this question. - From Review $\endgroup$
    – utobi
    Sep 7 at 10:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.