3
$\begingroup$

I came across explained variance score and $R^2$ score in scikit learn docs.

Docs defines exaplained variance score as:

$\text{explained variance} (y,\hat{y})=1-\frac{Var\{y-\hat{y}\}}{Var\{y\}}$

which I feel equates to $$1-\frac{\sum(y-\hat{y}-\overline{(y-\hat{y})})^2}{\sum(y-\bar{y})^2}\tag I$$

Docs then define $R^2$ score as follows:

$$R^2(y,\hat{y})=1-\frac{\sum(y-\hat{y})^2}{\sum(y-\bar{y})^2} \tag{II}$$

The difference between equation $\rm (I)$ and $\rm (II)$ is that equation $\rm (I)$ has $"-\overline{(y-\hat{y})}"$ in the numerator, while equation $\rm (II)$ does not. I have learnt from the various sources what different values of $R^2$ indicate:

  • $R^2<0→$ horizontal line explains the data better than the model, in other words, the chosen model does not follow the trend of the data
  • $R^2=0→$ horizontal line explains the data as well as the model, i.e. model has no explanatory value
  • $R^2>0→$ model explains the data better than the horizontal line
  • $R^2=1→$ means the model / regression line explains 100% of the variation in the response variable. This will be the case when all the data points lie precisely on the regression line

Q1. What to make sense of different values of "explained variance" and why?
Q2. Also, the doc points to this wiki page which does not give formula in equation $\rm (I)$. Why so?
Q3. How the two quantities are related intuitively?

$\endgroup$
2
  • 1
    $\begingroup$ See stackoverflow.com/questions/24378176/… $\endgroup$
    – Sergio
    Commented Jul 16, 2021 at 22:11
  • $\begingroup$ Put a lot time on that stackoverflow question. Accepted answer was quite convoluted in code. Also found second highest rated answer quite confusing. This is what I made sense from it and after some thinking, but need confirmation: Q3. "Explained variance" is unbiased version of $R^2$ obtained by subtracting $\overline{(y-\hat{y})^2}$ from the numerator of $R^2$. Q1. Since "explained variance" is just an unbiased version of $R^2$, its different values carry same meaning as explained in question for $R^2$. Q2. Not sure why. They might just have missed it. $\endgroup$
    – Rnj
    Commented Jul 19, 2021 at 11:47

1 Answer 1

1
$\begingroup$

The explained variance formula compares the variance of your residuals to the variance, which sounds great. However, the documentation says exactly why such a metric is not as helpful as it might appear.

The difference between the explained variance score and the R² score, the coefficient of determination is that when the explained variance score does not account for systematic offset in the prediction.

For instance, the explained variance function will see something like $y=(1,2,3,4)$ and $\hat y = (8, 9, 10, 11)$ and give a perfect score despite the predictions being awful. The formula $(II)$ would correctly flag this as problematic.

formula_1 <- function(y, yhat){
  ybar <- mean(y)
  return(
    1
    -
      var(y - yhat)
    /
      var(y - ybar)
  )
}
formula_2 <- function(y, yhat){
  ybar <- mean(y)
 return(
   1
   -
     sum((y - yhat)^2)
   /
     sum((y - ybar)^2)
 ) 
}
y <- c(1, 2, 3, 4)
yhat <- y + 7
formula_1(y, yhat) # I get 1, indicating a perfect fit
formula_2(y, yhat) # I get -38.2, indicating a terrible fit

This comes down to the fact that for any random variable $X$ with a finite variance and any $r\in\mathbb R$, $\text{var}(X) = \text{var}(X + r)$, yet $X = X+r$ does not hold in general, and this explained variance measure of performance does not catch such situations (which the sklearn $R^2$ function will catch).

Where the explained variance score could be useful is if you compare it to the $R^2$. If explained variance is good but $R^2$ is not, it could indicate systematic bias in the predictions, and you might be able to remedy this.

Note that the two can coincide, such as for in-sample predictions for an OLS linear regression.

set.seed(2023)
N <- 1000
formula_1 <- function(y, yhat){
  ybar <- mean(y)
  return(
    1
    -
      var(y - yhat)
    /
      var(y - ybar)
  )
}
formula_2 <- function(y, yhat){
  ybar <- mean(y)
  return(
    1
    -
      sum((y - yhat)^2)
    /
      sum((y - ybar)^2)
  ) 
}
x <- runif(N, -2, 2)
y <- x + rnorm(N) + 7
L <- lm(y ~ x)
yhat <- predict(L)
formula_1(y, yhat) # I get 0.5379571
formula_2(y, yhat) # I get 0.5379571
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.