I am reading up on linear regression from 18.650 MIT. In the way of explaining it, the professor derives a few results and I have attached them in the image enter image description here

  1. The first result gets used along with the theorem's mentioned towards the end in deriving the distribution of $\hat{\beta}$ which consequently gets used in testing the significance of the parameters
  2. The unbiased estimator of $\sigma^2$, the $\hat{\sigma}^2$ gets used in creating the prediction intervals (I have doubts about this. The professor did not mention this explicitly but, I think that is its purpose)


  1. What is the significance of the remaining three results? As one can see they depend upon $\sigma^2$ which is not known hence I cannot use them in their current form. These results were not used subsequently in deriving any other results as well
  2. Is the claim I have made above about the significance of the unbiased estimator of the variance true? (The unbiased estimator of $\sigma$, the $\hat{\sigma}$ gets used in creating the prediction intervals)

1 Answer 1

  1. $\sigma^2 tr((X^TX)^{-1}$ is telling us something about how difficult the regression problem is. First, $tr((X^TX)^{-1})$ is equal to the sum of the reciprocals of the eigenvalues of $X^TX$. So, if $X^TX$ is close to singular ($1/\lambda_i\rightarrow \infty$), we might have problems. This can occur with collinear columns in $X$. With this fixed design problem, we should choose a $X$ to avoid this. So, the difficulty of this problem is a function of the design of $X$ and the observation variance.

  2. $\sigma^2(n-p)$ is telling us how the problem is affected by the dimensionality of the task. Consider $n=p$ though, for each point, you may simply pass your regression through each point. This may minimize prediction error on your observations at the cost of overfitting. In a sense, we may always minimize the observed prediction error by increasing the number of predictors. Again, if $\sigma^2$ is large, then we will on average make more errors in our prediction.

  3. The theorem you have not discussed is used to derive the t-distribution for inference on $\hat{\beta}$, when $\sigma^2$ is unknown.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.