1
$\begingroup$

Suppose I have a linear regression model $$ \mathbf y = \mathbf X \mathbf b + \boldsymbol{\varepsilon},\,\boldsymbol{\varepsilon}\sim \mathcal N(\mathbf 0, \sigma ^2\mathbf I) $$ and use ML approach in order to estimate $\mathbf b$, that is: $$ \hat{\mathbf b}_{\mathrm {ML}} = (\mathbf X ^\top \mathbf X)^{-1}\mathbf X^\top \mathbf y $$ Consider residuals: $$ \mathbf r = \mathbf y - \mathbf X\hat{\mathbf b}_{\mathrm {ML}} = (\mathbf I - \mathbf X(\mathbf X ^\top \mathbf X)^{-1}\mathbf X^\top) \mathbf y \sim \mathcal N(\mathbf 0, \sigma ^2(\mathbf I - \mathbf P)^2), $$ where $\mathbf P = \mathbf X(\mathbf X ^\top \mathbf X)^{-1}\mathbf X^\top$.

How can I test normality of $\mathbf r$ using QQ- or PP-plots?

And another question: what is an unbiased estimate for $\sigma$ (since ML approach leads to biased estimate for $\sigma ^2$ as well as $\sigma$)?

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ As in one of the recent posts, also here I think it would be more common to use either $\hat \beta$ or $\text{b}$ in place of $\hat{\text{b}}$. $\endgroup$
    – Richard Hardy
    Commented Sep 25, 2015 at 14:56
  • $\begingroup$ I use C. Bishop's notation (except $\mathbf b$, while Bishop uses $\mathbf w$) for which $\beta$ stands for precision, i.e. inverse variance. However, thanks for advice, I will keep it in mind. $\endgroup$
    – vladkkkkk
    Commented Sep 25, 2015 at 15:31
  • 1
    $\begingroup$ How are you going to calculate $(I-P)^{-1}$? $\endgroup$
    – Glen_b
    Commented Sep 27, 2015 at 23:36
  • $\begingroup$ Oops, I have missed the fact that $\mathbf P$ is a projector, that is its inverse does not exist. Therefore, how can I test normality of $\mathbf r$ using QQ-plot? $\endgroup$
    – vladkkkkk
    Commented Sep 28, 2015 at 12:21

1 Answer 1

Reset to default
1
$\begingroup$

The components are independent, because the covariance of any two components is zero. You can see some more information about that here. I.e. having a diagonal covariance matrix implies that the covariance is zero, thus the individual components are independent.

Your second question is a bit more difficult. You can find an answer here, but explaining the details may require a bit longer answer.

Also when you say in the beginning that this is an ML approach, I do not completely agree. This is well established theory that is quite a lot older than ML, so I would rather refer to it as OLS, linear regression or linear model in general.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.