1
$\begingroup$

I think either i dont understand something or i try to mix something that are different things.

The mse value of a regression coefficients tells me how good i estimated the coefficent. Does it mean that the sum of all mse values of the regression coefficents is equal to the mse value of the whole regression model for prediction?

$\endgroup$
  • $\begingroup$ If you could elaborate...¿what do you mean the the MSE values of the regression coefficients? This seems to suggest you are running a number of bivariate regressions...¿maybe? $\endgroup$ – Gregg H Jun 6 '19 at 0:49
  • 1
    $\begingroup$ I think the OP means the mean squares due to $X_1$, $X_2$, etc. and comparing this to the mean squared error of the model. $\endgroup$ – StatsStudent Jun 6 '19 at 0:56
  • $\begingroup$ @GreggH StatsSTudent is right. The MSE of a regression coeffcients determines how biased the coefficients is and how bis his variance is. The mse of a prediction modell tells us the same in my opinion. But does this mean the MSE of a regression modell can be decomposed into the MSE values of the included regression coefficients of the regression model? $\endgroup$ – MasterStudent1992 Jun 6 '19 at 8:29
2
$\begingroup$

The variances of linear regression coefficient estimates aren't typically called "mean square errors" (MSE) but they are proportional to the model residual MSE as shown in this answer. What gets complicated is that the errors of the coefficient estimates are typically correlated.

Linear regression provides a symmetric variance-covariance matrix for the coefficient estimates:

$$ \widehat{\textrm{Var}}(\hat{\mathbf{\beta}}) = \hat{\sigma}^2 (\mathbf{X}^{\prime} \mathbf{X})^{-1}, $$ where $\hat{\sigma}^2$ is the residual MSE from the model and $\mathbf{X}$ is the design matrix representing the predictor values in the data set.

The estimated variances for the individual regression coefficients are the diagonal elements of this matrix. So the variance of each individual coefficient estimate is directly proportional to the residual MSE.

But if there are non-zero off-diagonal elements (covariances) in the matrix, then the variance of the sum of the coefficients isn't the same as the sum of their variances. You have to use the formula for the variance of a sum of correlated variables that takes the covariances into account. Say you have two positively correlated predictors. Each of their coefficients is likely to have a large variance but there will also be a substantial negative covariance between them so that the variance of their sum is less than the sum of their individual variances. In making predictions from the model based on those 2 predictors alone, the prediction error will be less than you would have expected from their individual coefficient variances. So summing the individual coefficient variances as you propose in the question will not provide easily interpretable results.

One final point, following up on one of your comments on the question: if errors are uncorrelated and have 0 mean and constant finite variance then standard linear regression coefficient estimates are not biased. Linear regression then provides the best linear unbiased estimates of the coefficients.

$\endgroup$
1
$\begingroup$

You can easily check with an example that that is not the case, even though they are somehow related (with all other things being equal, the larger the coefficient estimation error, the larger the mean-squared error of the model)

But this is not always the case. Think of a model like $Y:= 1 + 0.001*X + \epsilon$ where $\epsilon$ follows a $N(0, 10000)$ distribution. Now imagine you have a HUGE sample (like trillions of individuals) You will be able to get a good estimate for the model coefficients very accurately, but still, you won't get the mean-squared error below the variance of $\epsilon$

Of course, if our estimation error for the parameters were bigger, our MSE would be even greater, but you can't just say that the sum of the errors of the coefficients "adds up" to the mean-squared error, as the intrinsic variance of the output also plays a role

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.