I am currently reading a book "A first course in Linear model Theory". A linear regression model is defined as $$ y = X\beta+\epsilon$$ where $y = (Y_i, ... , Y_N )$ is an N-dimensional vector of observed responses,$\beta = ( \beta_0,\beta_1,...\beta_n)$'is a(k+1)-dimensional vector of unknown parameters,X is an $N*(k+1)$ matrix of rank r of known predictors, and $\epsilon= (\epsilon_1,\epsilon_2,...,\epsilon_n)$ is N-dimensional random vector of unobserved errors. Suppose,

$E(\epsilon)=0$ and $cov(\epsilon)=\delta ^2 I_N$ (1)

The book then writes :

Using (1) and the properties of the expectation and covariance operators:

$E(y)=E(X\beta+\epsilon)=X\beta+E(\epsilon)=X\beta$ (2)

$Cov(y)=Cov(X\beta+\epsilon)=Cov(\epsilon)=\delta ^2I_N$ (3)

I would appreciate so much a proof on how (1) and properties of expectation and covariance lead to equation (2) and (3) conclusions above


1 Answer 1


When $X$ and $\beta$ are assumed to be fixed, which is probably the case mentioned in the book, $X\beta$ term will be a constant inside the expressions. So, $$\operatorname{var}(\epsilon+c)=\operatorname{\epsilon}, \ \ \ E[\epsilon+c]=E[\epsilon]+c$$ Here, $c=X\beta$.

In a fully bayesian model, $X$ and $\beta$ could have been random as well. In that case, these expressions would have been $\operatorname{var}(y|X,\beta)$ and $E[y|X,\beta]$.

  • $\begingroup$ ${ x_1,...x_n}$ are random variables. can this be considered fixed. what disturbs me is $E(X\beta )=X\beta$ $\endgroup$ Feb 17, 2021 at 12:24
  • $\begingroup$ No, if they're assumed to be RVs in this analysis, the correct expression would be $E[X\beta|X]=X\beta$. Either they're assumed fixed in this particular analysis (not referring to another section of the book where $X$ can be assumed as random), or the expectation and variance notation is a bit slack. $\endgroup$
    – gunes
    Feb 17, 2021 at 12:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.