4
$\begingroup$

I have a binary classification problem where I apply logistic regression.

I have a set of features that are found significant.

But I understand that Logistic regression doesn't consider feature interactions

While I read online that, it can be accounted by adjusting logistic regression for con-founders

Currently I did this and got the significant features

model = sm.Logit(y_train, X_train)
result=model.fit()
result.summary()

But how do I adjust for confounders? any pointers to tutorial for non-stat person like me would really be helpful

Is there any python package that can help us do this?

Can someone let me know on how can I do this?

$\endgroup$

3 Answers 3

4
$\begingroup$

A non-statistically solid rule used in epidemiology is called "The 10% rule". It states that when the Odds Ratio (OR) changes by 10% or more upon including a confounder in your model, the confounder must be controlled for by leaving it in the model. If a 10% change in OR is not observed, you can remove the variable from your model, as it does not need to be controlled for.

EDIT:

What I think you are asking is:

1.) How to adjust for confounding via analysis

2.) If interaction is different than confounding.

An important thing to understand about confounding is that it is generally on a dataset by dataset basis. This works with the 10% rule. Essentially, if the OR of your exposure/outcome relationship does not change by 10% or more after adding the third variable into the model, there is not good enough evidence of confounding to keep it in the model. This is the case even if there is evidence in other literature that the third variable may be a confounder. You could hypothetically put in your model one exposure and 10 extra variables, but if none of the variables change the exposure/outcome relationship (OR), then you should not keep them in the model. Although leaving the variables in the model will control for them, but if it does not change your exposure/outcome relationship, it is unnecessary and will only cloud your conclusions. This article gives some other methods for addressing confounding, including the 10% rule (Just click on view PDF): Hernan 2002

Interaction differs from confounding in that it your exposure/outcome relationship is different on different levels of a third variable. Essentially, rather than the third variable influencing the OR like a confounder does, the third variable will have different ORs for different categories. An easy example is using gender (Male/Female). Say you are interested in the effect of cigarette smoking on cancer. You conclude that smoking gives an OR of 3.0 for cancer. However, you wish to examine if age is an interactive term. After entering the interactive term into your model, you find that the OR for cigarette smoking and cancer among males is 2.0 and among females is 4.0. In this case, reporting an OR of 3.0 would be misleading, as there is a clear difference between males in females.

Please let me know if this has clarified any issues you have.

$\endgroup$
4
  • $\begingroup$ Hi, thanks for the response. Upvoted. Exactly I understand the 10% rule. But what I am trying to do is listed in this post? datascience.stackexchange.com/questions/65600/… $\endgroup$
    – The Great
    Dec 29, 2019 at 23:56
  • $\begingroup$ Please see my edit. I hope this clarifies anything you may have been confused about. $\endgroup$ Jan 2, 2020 at 16:38
  • 1
    $\begingroup$ Do you have a citation for the 10% rule? $\endgroup$
    – Dave
    Jan 2, 2020 at 16:44
  • $\begingroup$ Thanks, just added one. $\endgroup$ Jan 2, 2020 at 16:49
6
$\begingroup$

The term confounder (or confounding) is used quite loosely these days. Usually, what people mean is to include additional variables into the regression model. In the medical literature, these are often age or gender. I'm not sure how this relates to interactions, which are merely used to allow for non-linear effects in a linear regression model. Consider the logistic regression model $$ \mathrm{logit}\left( \text{Diabetic} \right) = \beta_0 + \text{Weight} \cdot \beta_1 , $$ where the coefficient $\beta_1$ measures the contribution of weight ignoring a person's gender. When adding an interaction with gender, the model becomes $$ \mathrm{logit}\left( \text{Diabetic} \right) = \beta_0 + \text{Weight} \cdot I(\text{Gender} = \text{Male}) \cdot \beta_1 + \text{Weight} \cdot I(\text{Gender} = \text{Female}) \cdot \beta_2, $$ i.e., the contribution of weight is allowed to vary across genders. In contrast, "controlling for the confounder gender", would correspond to the model $$ \mathrm{logit}\left( \text{Diabetic} \right) = \beta_0 + \text{Weight} \cdot \beta_1 + I(\text{Gender} = \text{Male}) \cdot \beta_2 + I(\text{Gender} = \text{Female}) \cdot \beta_3 , $$ i.e., gender just becomes another feature, but it does not interact with weight and there is only a single coefficient for weight for males and females alike.

Defining Confounding

The term confounding is used loosely in the example above and it is not really clear how a confounder is different from any other feature.The causal inference literature defines confounding in mathematical terms. The objective is to estimate the causal effect of a set of variables $X$ on an outcome $Y$ by including a set other variables $S$ in the regression model to account for confounding. Variables in $S$ must satisfy the so-called back-door criterion. Consider the diagram taken from Pearl, 2009:

Example illustrating the back-door criterion

The back-door criterion has two conditions (Definition 3 in Pearl, 2009):

  1. No element of $S$ is a descendant of $X$
  2. The elements of $S$ "block" all "back-door" paths from $X$ to $Y$, namely all paths that end with an arrow pointing to $X$.

Using $S = \{Z_3\}$ would block the path $$X \leftarrow W_1 \leftarrow Z_1 \rightarrow Z_3 \rightarrow Y,$$ but not the path $$ X \leftarrow W_1 \leftarrow Z_1 \rightarrow Z_3 \leftarrow Z_2 \rightarrow W_2 \rightarrow Y , $$ because conditioning on the collider $Z_3$ opens the path between $Z_1$ and $Z_2$ and thus between $X$ and $Y$. Instead, we would need to include $S = \{Z_1, Z_2, Z_3\}$ in the regression model to control for confounding. Other sets are also possible, e.g. $S=\{W_1, Z_3\}$.

Unfortunately, this requires access to a graphical representation, as above, of how we imagine the world looks like. Often, this is unknown and people tend to include all variables in a regression model they have access to. However, there are certain situations where including additional variables in regression model actually introduces confounding, which otherwise would be absent.

$\endgroup$
3
  • $\begingroup$ Hi, thanks for the response. upvoted. Is there any implementation or python tutorial of this with an example data that you can share? would really be helpful $\endgroup$
    – The Great
    Dec 27, 2019 at 14:07
  • 1
    $\begingroup$ Accounting for confounding just means adding the confounders as additional columns to X_train in your example. The main problem is to determine which are the variables to include. This task cannot be solved programmatically, but must be based on your knowledge about the domain your data is coming from and is highly problem dependent. $\endgroup$
    – sebp
    Dec 27, 2019 at 14:16
  • $\begingroup$ Hi, yes. exactly. I now have the list of potential confounders based on domain knowledge and have included them in X_train as columns/features. Now does that mean my LR model is adjusted for confounders? Is that all? What I am trying to do is given in this post as well datascience.stackexchange.com/questions/65600/… $\endgroup$
    – The Great
    Dec 29, 2019 at 23:58
0
$\begingroup$

I have a question, @sebp:

Accounting for confounding just means adding the confounders as additional columns to X_train

  • Yes, but doesn't adding these additional columns also increase the dimensionality of your dataset?
  • Wouldn't that lead to potential overfitting of your predictive model by adding additional feature columns?
  • How do we balance the tension between the need to add these confounder columns with the curse of dimensionality by adding those same confounder columns?
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.