4
$\begingroup$

I'm studying linear regression and there is a concept I can't wrap my head around.

I've heard many times the expression "the independent variable explains $a$% of the variance of the dependent variable $y$. (where $a$ is a number)

The variance of the random variable $y$ is the distance of the observartions from the mean value of $y$. By adding our independent variable $x$ in the model, we want it to explain some of this variance.

I think that when we say the word explain, we don't actually mean that we explain something and it's a metaphorical way of using it but I'm not really sure what it means.

I've found another similar question here but the answer didn't help me understand it. I'd really appreciate some help!

Thanks

$\endgroup$
  • 1
    $\begingroup$ y is a random variable and we don't know its value with absolute certainty. without any additional information this uncertainty can be quantified by its variance. when we are given additional information like x, what percent of reduction in uncertainty is achieved? If we reach 100% then knowing x will mean knowing y precisely. This is what's meant by explained variance. $\endgroup$ – Cagdas Ozgenc May 13 '19 at 14:19
  • $\begingroup$ Hey, thanks for the help ! So let me get this straight. With no independent variables, our best guess for a prediction of $y$ is it's mean value, right? By adding a variable $x$ and making our model, we minimize the difference between the observation and the prediction, thus reducing the variance? My problem is, isnt variance the difference between a value and its mean? We dont actually reduce that, since the total variance (SST) always remains the same, doesnt it $\endgroup$ – Thomas May 13 '19 at 15:28
  • $\begingroup$ Our best guess for predicting a random value is it's mean value if your error function is for example squared loss. our prediction is affected by our objective. y is a random variable and it has a variance. y | x = x0 is another random variable (hopefully) with a lower variance. $\endgroup$ – Cagdas Ozgenc May 13 '19 at 15:40
  • $\begingroup$ Im sorry but isnt the variance always the same? The distance (squared) ok $y$ from it's mean value. By adding an x and making the regression line we dont have a new variance, we compertmentalize SST to SSR and SSE. I just didnt know i guess that we consider y-hat a new variable or something $\endgroup$ – Thomas May 13 '19 at 15:58
-1
$\begingroup$

I think what you are referring to is the definition of coefficient of determination (R-squared). It shows that how much variation in Y is explained by the variable X. For example if your r-squared is 0.86, it says that 86% of the variation in Y can be explained by X, while the rest is explained by some other variables not included in the model. Its used as a measure of goodness of fit of the model.

$\endgroup$
  • 1
    $\begingroup$ Thanks for the answer! Im aware of this, and of the $R^2$. The question was about this : What do we actually mean when we say that 86% of the variation in Y is explained by X? $\endgroup$ – Thomas May 13 '19 at 13:40
  • 1
    $\begingroup$ I downvoted as your answer is not what was asked. $\endgroup$ – James Phillips May 13 '19 at 17:16
  • $\begingroup$ it simply means the difference between the variance of Y variable and the variance of residuals. $\endgroup$ – Muzammil123 May 24 '19 at 14:00

Not the answer you're looking for? Browse other questions tagged or ask your own question.