# Confounding variables in machine learning predictions?

In classical statistics, confounding variable is a critical concept since it can distort our view about input variables and outcome variable's relationship. Many forms of control and adjustment are sought in statistics to eliminate, avoid or minimize the effect of confounding. For example, expected confounding variables (i.e., age and sex) are often included in the analysis, in the final model, the coefficient of your interested explanatory variable (i.e., treatment) is then adjusted for confounding (i.e., age and sex).

Confounding is not a frequent topic shows up in machine learning and predictive analysis. I wonder how confounding may (or may not) play an important role in machine learning algorithms. Does confounding potentially affect the accuracy of out-of-sample accuracy? Does including or not including an expected confounding variable play an important consideration when selecting as feature in machine learning?

Confounding plays a large role in statistics because we are looking to identify the exact effect of a set of variables on another. If confounding variables are left out of a statistical model then the effect measured for the variables that were included may be biased.

Confounding is not as a big a problem when performing prediction, because we are not concerned with identifying the exact effect of a variable on another. We are simply looking to find out what is the most likely' value of a dependent variable given a set of predictors.

So for example, suppose that we would like to estimate to what a degree a person's age is effects their salary. So we can estimate the model: $$\text{salary}_i = \beta_ 0 + \beta_1 \text{age}_i + \varepsilon_i.$$ It is very likely that $\beta_1$ in the equation above will be positive and fairly large, because older people tend to have more education and more work experience. So if we wish pin-point the link between age and salary, we should probably control for these confounders, estimating the model: $$\text{salary}_i = \beta_ 0 + \beta_1^* \text{age}_i + \beta_2 \text{education}_i + \beta_3\text{experience}_i + \varepsilon_i.$$ It is very likely that $\beta_1^* < \beta_1$ and that $\beta_1^*$ will be a much better estimator for the pure effect of age on one's earnings. That, in the sense of change someone's age and keep EVERYTHING else fixed'. However, since age is highly correlated with education and experience, the first model might just be good enough for predicting a person's salary.

• I do not know much about casual inference. But according to what I read here, education and experience could indeed influence salary but they are not cause of age, so they are not confounding variables per definition. Do I make a mistake understanding your answer? Oct 12, 2019 at 1:46

Confounding is a problem for prediction when the confounding relationship changes. This is a common problem for ML models in production. E.g., see What We Can Learn From the Epic Failure of Google Flu Trends.

The other common issue is feedback loops. eg Google presents advertising results based on predicting likelihood of response, but the position in the list affects the click through rate. Causality in machine-learning.

The only way to control for confounding is randomization because it will balance measured and unmeasured confounding. Any adjustment at the analytical level is only an attempt to minimize measured confounding (matching, restriction, G-methods...) but not eliminate it. So, if the goal of the exercise is merely prediction without an attempt to manipulate the outcome, which is the prediction goal, you don't need to care about confounding. But if you want to have a prediction model which you will use to change the outcome by manipulating predictors, then it is a much more challenging goal because, in that case, you need to adjust for confounders. The simplest example I can provide is: The increase in ice-cream sales will be an excellent predictor of an increased incidence of sunburns. However, if you want to change the incidence of sunburns banning the ice-creams sale will not help. Once you control for confounder (summer sun), the statistical connection between ice-cream sales is gone (because it is a statistical, not causal link). Now change the intuitive variables (ice-cream sales, sunburns, and warm weather) with non-intuitive (e.g. biomarker 1 in blood, risk of cancer, and the confounder that we don't even know that exist), and you will understand better why without randomization we are in a biased situation which is even hard to quantify in terms of magnitude but direction as well.

Confounding variables can result in quantities which are unrelated appearing to be correlated.

Imagine you want to check whether drinking red wine is good for your heart. You go out and survey a bunch of people to find out a) how much red wine they drink and b) some measure of cardiac health.

When you do this and plot the data on a graph, you might well find that there is a visible negative correlation. You then conclude:

$$$$\textbf{Wine}\implies \textbf{Cardiac Health}$$$$

Awesome! So drinking red wine makes your heart healthy, I should drink more it!

But wait, all we learned was that “People who drink more wine tend to have healthier hearts.”

What if income is a confounding variable and influences both wine consumption and health:

• Wealthy people tend to drink more wine.
• Wealthy people tend to have better healthcare.

To properly untangle these variables and figure out the true story we need to use causal inference (or causal discovery) to determine whether Wine consumption has an influence on cardiac health beyond the influence of personal wealth.

Depending on what you manage to learn via a causal approach to the confounders, you can try to constrain your ML algorithm to only use variables from your causal graph.

If you have a linear model, you can also constrain your model coefficients to be positive/negative and ensure your model is consistent with the discovered graph.