6
$\begingroup$

I am currently struggling on the meaning of the variance term $\sigma$ in the equation for computing the variance and the confidence interval of the mean reponse for a MLR: $$ Var[\hat{y}(x_0)]=\sigma^2\cdot x_0'(X'X)^{-1}x_0 $$ and $$ \hat{y}(x_0) - t_{\alpha /2, df(error)}\sqrt{\hat{\sigma}^2\cdot x_0'(X'X)^{-1}x_0} $$ $$ \leq \mu_{y|x_0} \leq \hat{y}(x_0) + t_{\alpha /2, df(error)}\sqrt{\hat{\sigma}^2\cdot x_0'(X'X)^{-1}x_0} . $$ (both formulas are taken from Myers, Montgomery, Anderson-Cook, "Response Surface Methodology" fourth edition, page 33-34)

1) Does $\sigma^2$ represents the variance of my data, or does it represent the variance of my Errors ?

Usually one uses the MSE as an estimator for $\sigma^2$, thus using the variance of my errors and not only of the data.

2) If I have already done a Measuring System Analysis (MSA), can I use the value of the calculated variance instead of the MSE of the Regression, because it would be a better estimator ?

$\endgroup$
4
$\begingroup$

MSE measures the variance of the error. To be clear -- that's the variance of the model errors, not the variance of the data. You can see this by looking at $SSE = (y_i - f(x_i))^2$. $SSE$ gives the squared difference between the observed and fitted values. Linear regression models are fit by minimizing $MSE$. From the Gauss-Markov theorem, we know that minimizing $MSE$ (i.e. using the "ordinary least squares" estimator) gives the best linear unbiased estimator of the coefficients, where "best" means the estimator with the lowest variance.

So the algorithm used to calculate an OLS regression model depends on the use of $MSE$, which estimates the variance of the model error (not the variance of the data), and gives the best linear unbiased estimator of the model coefficients (assuming the assumptions of regression, uncorrelated errors with expectation 0 and equal variance, are met). So there isn't a straightforward way to swap in a different estimate of variance, and it might not be what you're looking for anyway (again, variance of data vs. variance of model error). Furthermore, using a different approach will result in estimated coefficients no better than those obtained through OLS regression (based on MSE).

$\endgroup$
  • $\begingroup$ Thank you for your answer. I am aware that MSE represents the variance of my errors (due to the varaince of my data and the difference between observed and fitted value). After reading your answer, I was able to edit my question to stress out the point I don't understand. $\endgroup$ – John Tokka Tacos Jun 28 '18 at 12:48
  • $\begingroup$ When you compute the confidence interval of the regression response, you're giving a measure of the uncertainty of the estimates given by the model. So if you swap out the variance term for something that didn't come from the model, you're no longer giving a measure of the uncertainty of the estimates from the model. $\endgroup$ – djlid Jun 28 '18 at 12:55
  • $\begingroup$ But isn't the uncertainty due to the model already in the term $x_0'(X'X)^{-1}x_0 $ ? $\endgroup$ – John Tokka Tacos Jun 28 '18 at 12:58
  • 1
    $\begingroup$ I don't have a 100% definite answer for you. But what I'm trying to argue is the following. Let's say you want a 95% CI. If you calculate the CI using MSE -- and again, your model is producing estimates based on minimizing MSE, so using MSE matches the mechanism in your model -- you can feel pretty confident that you're getting a 95% CI. But if you swap your MSE-based estimate for a different estimate of $\sigma$, you're changing the width of your CI. Is it still a 95% CI? Probably not. Is it somehow "better?" Hard to say. I'd explore with a simulation study. Check the coverage probabilities. $\endgroup$ – djlid Jun 30 '18 at 12:11
1
$\begingroup$

The value $\sigma^2 = \mathbb{V}(\varepsilon_i)$ is the error variance in the regression model, and the variance result in your post is a consequence of the underlying variance of the OLS coefficient estimator:

$$\mathbb{V}(\hat{\boldsymbol{\beta}}) = \sigma^2 (\mathbf{x}^\text{T} \mathbf{x})^{-1}.$$

Since $\hat{y}(\mathbf{x}_0) = \mathbf{x}_0^\text{T} \hat{\boldsymbol{\beta}}$ you can use the ordinary rules for variances of random vectors to obtain

$$\mathbb{V}(\hat{y}(\mathbf{x}_0)) = \mathbb{V}(\mathbf{x}_0^\text{T} \hat{\boldsymbol{\beta}}) = \mathbf{x}_0^\text{T} \mathbb{V}(\hat{\boldsymbol{\beta}}) \mathbf{x}_0 = \sigma^2 \cdot \mathbf{x}_0^\text{T} (\mathbf{x}^\text{T} \mathbf{x})^{-1} \mathbf{x}_0.$$

The formula for the confidence interval then follows from the following pivotal quantity:

$$\frac{\hat{y}(\mathbf{x}_0) - \mu_0}{\hat{\sigma}/df_{Res}} \sim \text{Student's T}(df_{Res}),$$

where $\mu_0 = \mathbb{E}(y(\mathbf{x}_0))$ and $\hat{\sigma}$ is the standard bias-corrected MLE in the linear regression. Now, there is no particular reason you could not substitute this with a different estimator for $\sigma$ if you want to, but bear in mind that it could change the distribution of the pivotal quantity you are using to form your confidence interval. So the thing you would need to do if you want to substitute a different estimator is to see how this would affect the distribution of the newly created quantity. Many variance estimators have an asymptotic chi-squared distribution (via the CLT) so you might end up with the same distribution, and hence the same form for your confidence interval, but you should still check this.

$\endgroup$
  • $\begingroup$ @ Ben thanks for your answer ! The difference between the two estimators is: 1. MSE represents the variance due to the data variance (SSPE) AND the model errors (SSLoF). 2. $\sigma^2$ taken from the MSA contains only the variance in the data due to the precision of my measuring device. After a quick check, I saw that the variance of the Pure Error term ($SSPE/(n-m)$) is very similar to the variance of the MSA. The MSE of my Regression is bigger than both, because it adds the variance due to my lack of fit. I will therefore use MSE as estimator because it is more "critical" ! $\endgroup$ – John Tokka Tacos Jul 2 '18 at 9:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.