I read a paper and it says "we regressed out the effect of a variable via linear regression".

Can someone provide to recipe to remove an effect of both numerical and categorical variables?

For example, remove the age effect: I know that my variable of interest depends on age and I want to remove that dependency.

Remove the site/measurement effect: I know that my variable of interest depends on the site of measurements -- on one site the variable is underestimated, on other it's overestimated, and I want to remove that dependency.

How I may act in the latter case -- I would split the whole data into the subpopulations depending on the sites, calculate overall mean of the whole data, and the means of the subpopulations and will add the difference between the overall mean and the subpopulation mean to the each subpopulation.

But how it's related to "regress out" and linear regression? For me a linear regression is just a line you can fit to the data, how one can relate the process of removing effect to that line and give a geometric intution? And how to deal with numerical data where I can't break the data into subpopulations?


2 Answers 2


The power and utility of linear regression (or regression analysis in general) is not just that it fits the best line through a bunch of data. Rather, regression analysis has the ability's to estimate the relationship between an independent and dependent variable "holding other variables constant." (Saying you "regressed out" the effect of a variable is an odd way to phrase it, usually we say "we controlled for variable X using a regression model.")

It sounds like you are only thinking of a linear regression (also known as OLS - ordinary least squares) model with a single independent variable, which amounts to estimating the intercept and slope of the line of best fit (the line that minimizes the sum of squared errors) through a 2 dimensional cloud of points. But an OLS models can include multiple independent variable. Lets say it includes two (variables A and B). The model is

$Y = \beta_0 + \beta_1*A + \beta_2*B + \epsilon$

Here, instead of trying to fit a line through a 2D cloud of points, he model is trying to fit a plane through a three dimensional cloud of points. And that plane has two "partial slopes." The partial slope for A (\beta_1*) tells you what happens to Y when you increase A by one but hold B constant. The partial slope for B tells you what happens to Y when you increase B by 1 but hold A constant.

Obviously this can be extended to more variables. It is harder to visualize because you end up talking about four or five dimensional spaces, but the math is the same.

This is how an OLS model "controls for" different variables. Critically, this works just as well regardless of whether the independent variables are continuous (like age) or binary (like "treatment" vs "control"). As long as it makes sense to say that the variable "increased by one" the math works out. You do run into complications if you are interested in a NOMINAL categorical variable with multiple categories that are not in any sort of order (like being in site A, B, C or D). For those types of variables "increasing by one" doesn't mean anything so the model would give you garbage. The solution is to split that variable into a series of binary "dummy" variables. Se here for more on that - although that question is about a different kind of regression (logit) the issues for how you set up independent variables are the same.


It's hard to know for sure what the authors meant; maybe other bits of the article would offer more clues. But what I think this means is to do a regression and then remove the effect of one variable. So, a regression yields something like:

$Y = \beta_0 + \beta_1x_1 + \beta_2x_2 + \epsilon$

then if they "regressed out" the effect of $x_1$ they would get:

$AdjY_i = Y_i + \beta_1x_{1i)$

for each observation.

Of course, it would be more complex if there were interactions or quadratic effects or whatever, but the idea would be the same.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.