I would like to ask a question regarding prediction based on logistic regression model.

At my work we are trying to predict if an employee is going to leave in the next period. We want to utilize logistic regression for this. Let’s assume we have developed a model based on both continuous (e.g. salary, age etc ) and categorical (e.g. city, department etc – using dummies) independent variables. Let’s assume the model has good adjusted R squared and does not suffer from multicollinearity . For the sake of simplicity let’s assume we have developed regression formula where probability is a function of city (categorical) and salary (continuous): p=f(const + b1*city_1 + b2*city_2 + … + bn*salary )

My specific question is this. How can I decide which of the factor has the biggest contribution to the fact, that an employee's risk of leaving is high? Is it in this case a city or the salary for the particular employee? And what about if the model is more complex than this containing both categorical and continuous variables and I want to list maybe three the most important factors? I fear it is not possible to only compare b1*city_1 whith b2*city_2 and bn*salary because of what happens if the value for the given employee is city_3 which is represented by the constant due to the k-1 rule for dummies?

I know my question might sound a bit not structured. Thanks a lot for you advice and effort in advance.


Logistic regression has the nice feature that the odds remain the same irrespective of the base probability. So the odds $e^{\beta_\mathrm{salary}}$ (more precisely this is the multiplicative odds ratio relative to the baseline odds $e^{\beta_0}$ per unit of salary) are the same whether you consider city 1 or city 2. Conversely you can consider the odds of leaving say city 1 relative to the baseline city 0 irrespective of the salary level. (Though consider that the expected salary level may vary across cities).

When interpreting odds ratios, recall that odds are strictly positive numbers with values greater than one indicating odds increase. Odds can be difficult to interpret and Gelman and Hill (2006) provide some alternative ideas on interpreting the coefficients of logistic regression in chapter 5.2.

Since money and salary and their units are familiar to every adult and in business context they are a likely candidate for intervention, it is useful to normalize other regression coefficients in terms of amount of money. Formally, we are evaluating a contra-factual causal quantity: all things being equal, what is the average amount of money, needed to match the change caused by some other non-monetary factor. For instance, if the probability of leaving is higher in city 1 than in city 2, one may ask what additional amount of money is required to pay each employee in city 1 so that the probability of leaving is the same in the two cities. This is given by $(\beta_1-\beta_2)/\beta_n$ in units of salary. Similar strategy can be used to compare across factors in terms of salary costs. Just note that this quantity depends on the baseline probability of leaving, so if, say, the average salary increases for the next period this quantity won't provide an accurate prediction.

Gelman, A., & Hill, J. (2006). Data analysis using regression and multilevel/hierarchical models. Cambridge university press.

| cite | improve this answer | |

First, you can find a really good explanation http://www.appstate.edu/~whiteheadjc/service/logit/intro.htm

On general matters. Logistic regression models assign a weight to each variable. This weight has a real value, positive, zero or negative.
Since the model is linear - of the form $$ \alpha_{0}*1 + \alpha_{1}*\beta_{1} + \alpha_{2}*\beta_{2} + .. $$ each variable affect on the end result can be calculated independently of others.

How can you compare variables weights? and decide which one has the strongest effect on outcome probability?

To answer this question there are three things you need to consider:
1) Base model odds ratio.
2) Variable weight.
3) Variable distribution.

Based model odds ratio:
When you train a model with a slope(intercept) you basically assume the starting point is not necessary 0. In your case the probability of an employee to leave, might be positive regardless of all variables.
This has other importance in the case of categorical variables.
The intercept factor ($\alpha_{0}$) states what is the odds ratio of sample having all variables equal to zero. In your case an employee living in the one city you didnt represent in its own variable.

By agreeing on the fact that the intercept value is actually a weight of the default-state variable its fair comparing it to other variables.

The variable weight
Assuming weight $\alpha_{i}$, taking its exponent $e^{\alpha_{i}}$ we get the numerical effect of one unit change in the corresponding variable $\beta_{i}$ on the odds ratio (P(1)\(1-P(1))).
Be aware that some weights might be negative and an increase in their corresponding variable will result in lower odds ratio.
Lets assume that we $e^{\alpha_{1}}=2$ and that $e^{\alpha_{2}}=0.5$. For this case, an increase in one unit in $\beta_{1}$ will increase the odds ratio by 2 and an increase in one unit in $\beta_{2}$ will reduce the odds ratio by 2. So we might say both variables have the same effect strength, only in a different direction.
Now you can compare each variable effect strength.

Variable Distribution
There is a lot of math behind this part, but asking this two step question might help avoiding assuming importance where there is none.
For each variable, how many times do i expect to see it change in more than one unit? and how much more? If some of the employees are paid extremely high than other and they quit their job more often than other than looking at the weight of variable salary might be deceiving. Reason being that for all the salary values between normal employees salaries and extremely high salary employees we have no data(and our model assumes continuity).

There are statistic tests for getting a technical measure of the weight reliability(Wald test for example).

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.