1
$\begingroup$

I don't understand linear regression.

Assume the classic linear model:

$$Y = X \beta + \epsilon,\\ \epsilon \sim \mathbb{N}(0, \sigma^2 I_n), $$ where $Y$ is a vector of length $n$, $X$ is a matrix of size $n \times p$. How it's possible that $\epsilon$ is indeed distributed with a $\sigma^2 I_n$ covariance matrix? In my understanding that would imply that there is $n$ orthogonal 1-dimensional vectors, which is a nonsense, of course. On the other hand, consider the equation $$Y = X \beta + \mathbb{1} \epsilon,\\\epsilon \sim \mathbb{N}(0, \sigma^2),$$ where $\mathbb{1}$ is a vector of ones of length $n$. This would imply in turn a $\sigma^2 \mathbb{1} \mathbb{1}^T$ covariance matrix, which makes sense to me as it has only 1 non-zero eigenvalue.

What I am missing here?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Re "that would imply:" you misinterpret the notation. There are $n$ orthogonal $n$-vectors; namely, $(\epsilon_1,0,\ldots,0),$ $(0,\epsilon_2,0,\ldots,0),$ and so on. $\endgroup$
    – whuber ♦
    Jul 29, 2020 at 14:09
  • $\begingroup$ Whuber, thank you. I feel idiotic now. Wish I could upvote you. $\endgroup$
    – Geo
    Aug 1, 2020 at 12:26

1 Answer 1

Reset to default
1
$\begingroup$

In your first equation $\epsilon \sim \mathbb{N}(0, \sigma^2 I_n)$ is a random vector with different elements. Note that it's an assumption of homoskedasticty, that is "different values of the response variable have the same variance in their errors, regardless of the values of the predictor variables" (from wikipedia) and of no autocorrelation between observations.

You can also assume $\epsilon \sim \mathbb{N}(0, \Sigma)$, where $\Sigma$ is a positive semi-definite covariance matrix. This weaker assumption affects the choice of the estimator since, for example, in the first case (together with other assumptions) the OLS estimator is the best one in terms of variance, while in the latter case you can find better estimators.

In the latter you are assuming the same error $\epsilon \sim \mathbb{N}(0, \sigma^2)$ for every observation, which is a way stricter assumption.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Yes, that explains why the second equation is not a great practical choice. I think I've included it just to help explain my problem with understanding the $I_n$ assumption. It looks paradoxical to me in a sense that if I, let's say, have actual observations for $\epsilon_i$, there is no way I can get anything proportional to $I_n$ from their products. $\endgroup$
    – Geo
    Jul 29, 2020 at 12:33
  • $\begingroup$ I edited the answer hoping to explain why one could assume $\sigma^2I_n$. Also, in the above comment you are talking about observing $\epsilon_i$, this is not possible as it's unobserved. You can estimate $\beta$ and compute the residual $e_i = y_i - x_i'\hat{\beta}$. $\endgroup$
    – Ale
    Jul 29, 2020 at 12:57
  • $\begingroup$ What I really meant by saying "observations of $\epsilon$" is a realization of random variable $\epsilon$. If I generate synthetic data for the linear model, I first fill predictors with some random values, then fill the response variable from them, and then, finally, I'll add some "errors" to the generated data. Those errors can be sampled from $\mathbb{N}(0, \sigma^2 I_n)$, for example. But if I compute the sample covariance matrix for them, it will never be $\sigma^2 I_n$. In fact, I could have sampled them from any other distribution and still get the same covariance matrix by chance. $\endgroup$
    – Geo
    Jul 29, 2020 at 13:13
  • $\begingroup$ But know you are talking about the difference between an assumption on the process generating the data and what you observe from a sample. $\endgroup$
    – Ale
    Jul 29, 2020 at 14:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.