Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
In this question linked below, it was addressed why we would center the features in linear regression.
What I understood was that its because it gave the bias term meaning - it made it the predicted value of the label when all the features were at their means.
If $y$ is the value of the label, it gave the intercept with the $y$ axis meaning.
However, would there ever be any reason for centering the labels? What would that even mean, graphically and intuitively?
Thanks!
Say you have a dataset with 2 features, $X$, and $Y$.
Centering the features of this dataset would mean we make it so that the mean value of the new $X$ feature becomes $0$, and the mean value of the new $Y$ feature becomes $0$.
In other words, if we get a dataset where the $X$ and $Y$ feature-values are centered, that means that they're given in terms of the distance from the most average feature-values for that dataset. When $X$ is $0$ and $Y$ is $0$, that corresponds to the "most average" datapoint.
Now, let's say we want to predict $Y$ based on $X$ by doing linear regression on this data-set. In other words, we are turning $Y$ into the label we want to predict, and $X$ into the single feature we're predicting it with.
Now, what happens? If we don't center either of the random-variables, then our linear regression model will simply fit a line to the $Y_s$ based on the $X_s$. $$y=mx+b$$.
That's okay. However, in this case $+b$, the intercept, has absolutely no meaning. It's the label which our model predicts a datapoint would take when $X=0$. However, $X$ might correspond to a feature where equaling zero is absolutely senseless! For example, what if $X$ is the height of a newborn baby?
By centering $X$, the bias now has meaning. Since $X$ is centered, it now takes on a value of $0$ for its average value. That makes the bias, $+b$, equal to the $Y$ feature-value for the baby with "the most average height!!!" Pretty cool!
Now, what if we center the $Y$ labels as well? What happens to the bias - what does it represent?
First of all, notice what happens to the linear if we center the $Y$ labels. Centering the $Y$ labels means that we subtract the mean of $Y$ from each $y$ value, so that the value that was previously the mean.
Let $X'$ be the centered feature, $Y$ be the uncentered label, and $Y'$ be the centered label.
If all we're doing is setting $Y'=Y-\mu_y$, then our linear equation should go from...
$$y=mx'+b$$
...to...
$y'=mx'+(b-\mu_y)$
...that is, all that should happen to our linear equation is that it should get shifted down by $\mu_y$.
Say that $Y$ corresponds to the weight of a newborn baby.
Before centering $Y$ and turning it into $Y'$, when $X'=0$ the linear equation told us what we should predict the weight of a newborn baby of average height to be. That was $b$.
Now, when $X'=0$, we instead get $b-\mu_y$. That is, we get the expected difference between the weight of a newborn baby of average height and the average weight of these babies!
Now, for each value of $X'$, what we're getting is the expected difference between the weight of a baby whose height is $X'$ away from the average height and the weight of the baby with the average weight!
When we fit a regression, we are trying to explain variance in our dependent variable. That is, we are trying to explain why the datapoints vary around the mean value. If we mean centre our dependent variable, we are just setting the mean to 0, but not changing anything else about it. If a person's height is 10cm above the mean, that won't change if I subtract 150cm from everyone's heights. So it seems intuitive that the model shouldn't change.
R code to show it doesn't change:
We fit a multiple regression to the popular dataset built into R: 'mtcars'. The model predicts a car's miles per gallon from a few things about that car.
my_multiple_regression <- lm(mpg ~ wt + qsec + am , mtcars)
summary(my_multiple_regression)
Now we make a new variable, mpg_mean_centred, and refit the regression on this:
mtcars$mpg_mean_centred = mtcars$mpg - mean(mtcars$mpg)
my_multiple_regression2 <- lm(mpg_mean_centred ~ wt + qsec + am , mtcars)
summary(my_multiple_regression2)
If you run the code you'll see the regression outputs are exactly the same
