A recipe for removing confounding variable effect

Question

I read a paper and it says "we regressed out the effect of a variable via linear regression".

Can someone provide to recipe to remove an effect of both numerical and categorical variables?

For example, remove the age effect: I know that my variable of interest depends on age and I want to remove that dependency.

Remove the site/measurement effect: I know that my variable of interest depends on the site of measurements -- on one site the variable is underestimated, on other it's overestimated, and I want to remove that dependency.

How I may act in the latter case -- I would split the whole data into the subpopulations depending on the sites, calculate overall mean of the whole data, and the means of the subpopulations and will add the difference between the overall mean and the subpopulation mean to the each subpopulation.

But how it's related to "regress out" and linear regression? For me a linear regression is just a line you can fit to the data, how one can relate the process of removing effect to that line and give a geometric intution? And how to deal with numerical data where I can't break the data into subpopulations?

Answer 1

For a simple (continuous) example suppose you have a dataset $\{(x_{i1},x_{i2},y_i)\}_{i=1}^{n}$ of two independent variables where $x_1$ is a relevant variable and $x_2$ is a confounding variable.

Given the linear model $$ \hat y = \beta_0 + \beta_1x_1 + \beta_2x_2 + \epsilon_i $$ you estimate the coefficients $\beta_0,\beta_1,\beta_2$ using least-squares regression. Then the "corrected" value of the $i$-th datapoint is given by $y_i^* = y_i - \beta_2x_{i2}$.

This can be done this in R using the limma package, or by hand like in this blog post.

How I may act in the latter case -- I would split the whole data into the subpopulations depending on the sites, calculate overall mean of the whole data, and the means of the subpopulations and will add the difference between the overall mean and the subpopulation mean to the each subpopulation.

This is exactly what happens in the case when the confounding variable is discrete.