1
$\begingroup$

I have a dilemma with respect to the included (decomposition) between bias and variance in the calculation of the Mean square error (MSE) for the OLS estimator with the equation: MSE = bias ^ 2 + variance

I calculated with R software the bias, the variance and the MSE. As you will see I run the code many times (replications = 1000 times).

And it is clear that MSE is not equal to bias ^ 2 + variance. However, I do not think I was wrong in the 3 formulas (bias, variance and MSE) neither in my R code in general. Since equality is not respected, can I then deduce a strong/extra noise in the data ? If not, how to explain this inequality ?

You can copy and paste the code below.

###########################
# Data
PIB.hab<-c(12000,34000,25000,43000,12500,32400,76320,45890,76345,90565,76580,45670,23450,34560,65430,65435,56755,87655,90755,45675)
ISQ.2018<-c(564,587,489,421,478,499,521,510,532,476,421,467,539,521,478,532,449,487,465,500)

Dataset=data.frame(ISQ.2018,PIB.hab)

#plot
plot(ISQ.2018,PIB.hab)
plot(ISQ.2018,PIB.hab, main="Droite de régression linéaire", xlab="Score ISQ 2018", ylab="PIB/hab")

#OLS fit
fit1<-lm(PIB.hab~ISQ.2018)
lines(ISQ.2018, fitted(fit1), col="blue", lwd=2)

# Create a list to store the results
lst<-list()

# This statement does the repetitions (looping)

for(i in 1 :1000)
{

 n=dim(Dataset)[1]
 p=0.667
 sam<-sample(1 :n,floor(p*n),replace=FALSE)
 Training <-Dataset [sam,]
 Testing <- Dataset [-sam,]
 fit2<-lm(PIB.hab~ISQ.2018)
 ypred<-predict(fit2,newdata=Testing)
 y<-Dataset[-sam,]$PIB.hab
 MSE <- mean((y-ypred)^2)
 biais <- mean(ypred-y)
 variance <-mean((ypred- mean(ypred))^2)

 lst[[i]] <- c(MSE = MSE,
                    biais = biais,
                    variance = variance)
 # lst[i]<-MSE
 # lst[i]<-biais
 # lst[i]<-variance

}

# convert to a matrix

x <- as.matrix(do.call(rbind, lst))
colMeans(x)
########################### 
$\endgroup$
1
  • 1
    $\begingroup$ Bias in the bias-variance decomposition should be squared. But that is not the main problem here. On any empirical dataset, you cannot decompose the MSE into bias, variance and irreducible error. For that, you need to know the true data-generating model, that is, f(x) in the formula in the post of @AHK. $\endgroup$ Feb 21, 2021 at 21:51

1 Answer 1

1
$\begingroup$

If for some observed data $(X,Y)$ we have a model $Y = \hat{f}(x) + \epsilon$, the bias-variance trade off is given by:

$$ \mathbb{E}\left[(Y - \hat{f}(X))^2\right] = \left(\mathrm{bias}(\hat{f}(X))^2 \right) + \mathrm{var}(\hat{f}(X)) + \sigma_\epsilon^2 $$ where $\mathrm{bias}(\hat{f}(X)) = \mathbb{E}\left[\hat{f}(X) - f(X)\right]$ and $\mathrm{var}(\hat{f}(X)) = \mathbb{E}\left[\left(\hat{f}(X) - \mathbb{E}\left[\hat{f}(X)\right]\right)^2\right] $. The third term is irreducible error.

In the case of linear regression we have the true model is $f(x) = \beta_0 + \beta_1 X$ and the predicted value for $Y$ is $\hat{f}(X) = \hat{\beta}_0 + \hat{\beta}_1 X = \hat{y}$.

In general, the bias variance tradeoff is a statement about the distribution, and may not hold for a finite sample.

That being said the following equality must hold in OLS: $$ \sum_{i=1}^{n}( y - \bar{y})^2 = \sum_{i=1}^{n} (y - \hat{y})^2 + \sum_{i=1}^{n} (\bar{y} - \hat{y})^2 $$ However, this is not what is usually referred to as the “bias-variance tradeoff”

For more details see this link

$\endgroup$
4
  • $\begingroup$ Hi AHK, many thanks for your complete response. So, if I correctly understand your answer, my R code is OK and if the equality (MSE = bias^2 + variance) does not hold it may be because of the epsilon square (extra noise) in your 1st equation and/or maybe because I calculate the bias-variance tradeoff in a finite sample. Is it correct ? $\endgroup$ Jan 31, 2021 at 16:10
  • $\begingroup$ My guess is yes, but make sure that your code gets the equality in the second equation $\endgroup$
    – blooraven
    Feb 1, 2021 at 16:38
  • $\begingroup$ Thanks for the edit @marjolein $\endgroup$
    – blooraven
    Feb 21, 2021 at 22:12
  • $\begingroup$ @AHK you're welcome! Forced me to take a closer look at the formula, always helpful for improving my own understanding. $\endgroup$ Feb 21, 2021 at 22:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.