I have a binary classification problem where I apply logistic regression.

I have a set of features that are found significant.

But I understand that Logistic regression doesn't consider feature interactions

While I read online that, it can be accounted by adjusting logistic regression for con-founders

Currently I did this and got the significant features

model = sm.Logit(y_train, X_train)

But how do I adjust for confounders? any pointers to tutorial for non-stat person like me would really be helpful

Is there any python package that can help us do this?

Can someone let me know on how can I do this?


A non-statistically solid rule used in epidemiology is called "The 10% rule". It states that when the Odds Ratio (OR) changes by 10% or more upon including a confounder in your model, the confounder must be controlled for by leaving it in the model. If a 10% change in OR is not observed, you can remove the variable from your model, as it does not need to be controlled for.


What I think you are asking is:

1.) How to adjust for confounding via analysis

2.) If interaction is different than confounding.

An important thing to understand about confounding is that it is generally on a dataset by dataset basis. This works with the 10% rule. Essentially, if the OR of your exposure/outcome relationship does not change by 10% or more after adding the third variable into the model, there is not good enough evidence of confounding to keep it in the model. This is the case even if there is evidence in other literature that the third variable may be a confounder. You could hypothetically put in your model one exposure and 10 extra variables, but if none of the variables change the exposure/outcome relationship (OR), then you should not keep them in the model. Although leaving the variables in the model will control for them, but if it does not change your exposure/outcome relationship, it is unnecessary and will only cloud your conclusions. This article gives some other methods for addressing confounding, including the 10% rule (Just click on view PDF): Hernan 2002

Interaction differs from confounding in that it your exposure/outcome relationship is different on different levels of a third variable. Essentially, rather than the third variable influencing the OR like a confounder does, the third variable will have different ORs for different categories. An easy example is using gender (Male/Female). Say you are interested in the effect of cigarette smoking on cancer. You conclude that smoking gives an OR of 3.0 for cancer. However, you wish to examine if age is an interactive term. After entering the interactive term into your model, you find that the OR for cigarette smoking and cancer among males is 2.0 and among females is 4.0. In this case, reporting an OR of 3.0 would be misleading, as there is a clear difference between males in females.

Please let me know if this has clarified any issues you have.

  • $\begingroup$ Hi, thanks for the response. Upvoted. Exactly I understand the 10% rule. But what I am trying to do is listed in this post? datascience.stackexchange.com/questions/65600/… $\endgroup$ – The Great Dec 29 '19 at 23:56
  • $\begingroup$ Please see my edit. I hope this clarifies anything you may have been confused about. $\endgroup$ – coconn41 Jan 2 at 16:38
  • 1
    $\begingroup$ Do you have a citation for the 10% rule? $\endgroup$ – Dave Jan 2 at 16:44
  • $\begingroup$ Thanks, just added one. $\endgroup$ – coconn41 Jan 2 at 16:49

The term confounder (or confounding) is used quite loosely these days. Usually, what people mean is to include additional variables into the regression model. In the medical literature, these are often age or gender. I'm not sure how this relates to interactions, which are merely used to allow for non-linear effects in a linear regression model. Consider the logistic regression model $$ \mathrm{logit}\left( \text{Diabetic} \right) = \beta_0 + \text{Weight} \cdot \beta_1 , $$ where the coefficient $\beta_1$ measures the contribution of weight ignoring a person's gender. When adding an interaction with gender, the model becomes $$ \mathrm{logit}\left( \text{Diabetic} \right) = \beta_0 + \text{Weight} \cdot I(\text{Gender} = \text{Male}) \cdot \beta_1 + \text{Weight} \cdot I(\text{Gender} = \text{Female}) \cdot \beta_2, $$ i.e., the contribution of weight is allowed to vary across genders. In contrast, "controlling for the confounder gender", would correspond to the model $$ \mathrm{logit}\left( \text{Diabetic} \right) = \beta_0 + \text{Weight} \cdot \beta_1 + I(\text{Gender} = \text{Male}) \cdot \beta_2 + I(\text{Gender} = \text{Female}) \cdot \beta_3 , $$ i.e., gender just becomes another feature, but it does not interact with weight and there is only a single coefficient for weight for males and females alike.

Defining Confounding

The term confounding is used loosely in the example above and it is not really clear how a confounder is different from any other feature.The causal inference literature defines confounding in mathematical terms. The objective is to estimate the causal effect of a set of variables $X$ on an outcome $Y$ by including a set other variables $S$ in the regression model to account for confounding. Variables in $S$ must satisfy the so-called back-door criterion. Consider the diagram taken from Pearl, 2009:

Example illustrating the back-door criterion

The back-door criterion has two conditions (Definition 3 in Pearl, 2009):

  1. No element of $S$ is a descendant of $X$
  2. The elements of $S$ "block" all "back-door" paths from $X$ to $Y$, namely all paths that end with an arrow pointing to $X$.

Using $S = \{Z_3\}$ would block the path $$X \leftarrow W_1 \leftarrow Z_1 \rightarrow Z_3 \rightarrow Y,$$ but not the path $$ X \leftarrow W_1 \leftarrow Z_1 \rightarrow Z_3 \leftarrow Z_2 \rightarrow W_2 \rightarrow Y , $$ because conditioning on the collider $Z_3$ opens the path between $Z_1$ and $Z_2$ and thus between $X$ and $Y$. Instead, we would need to include $S = \{Z_1, Z_2, Z_3\}$ in the regression model to control for confounding. Other sets are also possible, e.g. $S=\{W_1, Z_3\}$.

Unfortunately, this requires access to a graphical representation, as above, of how we imagine the world looks like. Often, this is unknown and people tend to include all variables in a regression model they have access to. However, there are certain situations where including additional variables in regression model actually introduces confounding, which otherwise would be absent.

  • $\begingroup$ Hi, thanks for the response. upvoted. Is there any implementation or python tutorial of this with an example data that you can share? would really be helpful $\endgroup$ – The Great Dec 27 '19 at 14:07
  • $\begingroup$ Accounting for confounding just means adding the confounders as additional columns to X_train in your example. The main problem is to determine which are the variables to include. This task cannot be solved programmatically, but must be based on your knowledge about the domain your data is coming from and is highly problem dependent. $\endgroup$ – sebp Dec 27 '19 at 14:16
  • $\begingroup$ Hi, yes. exactly. I now have the list of potential confounders based on domain knowledge and have included them in X_train as columns/features. Now does that mean my LR model is adjusted for confounders? Is that all? What I am trying to do is given in this post as well datascience.stackexchange.com/questions/65600/… $\endgroup$ – The Great Dec 29 '19 at 23:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.