1
$\begingroup$

Let $X$ be an $n\times p$ full-rank matrix of predictors with $n > p$, and $y$ be the vector of responses. Assume $y \sim N(X\beta, \sigma^2I)$, and let $\hat{\beta}$ be the least-squares estimator of $\beta$. Let $e = y - X\hat{\beta}$ the vector of residuals.

Since $y$ is normal, then $\hat{\beta} = (X^TX)^{-1}X^Ty \sim N(\beta, \sigma^2(X^TX)^{-1})$. And $E(\hat{\beta}) = \beta$.

What does it mean to take the conditional expectation given the residuals? i.e., $E(\hat{\beta}|e = e_0),$ where "=" is interpreted component-wise.

I have two questions:

  1. Intuitively, what exactly does it mean to condition on the residuals $e = e_0$? Since residuals $e = y - X\hat{\beta}$, does this mean that we're assuming that $y = y_0$ and $\hat{\beta} = \hat{\beta}^*$ such that $e_0 = y_0 - X\hat{\beta}^*$? That is, we are conditioning on a specific vector of realizations for the responses $y_0$ and a specific vector of OLS coefficients $\hat{\beta}^*$?
  2. If the above were true, I thought the probability of a continuous random variable is 0. Therefore, is the probability of $e = e_0$ also 0 since $e$ are continuous?
$\endgroup$

1 Answer 1

0
$\begingroup$

Question 1

If $y\sim\mathcal{N}(X\beta, \sigma^2I)$, you can obtain $e$ via a linear transformation of $y$, $e=(I-H)y$, this means $E$ is normally distributed with mean $(I-H)X\beta$ and covariance $\sigma^2(I-H)(I-H)^T$. Now you have the pdf $f_E(e)$ and its value at some $e_0$.

You need to estimate the joint distribution $f(\hat{\beta}, E)$. This can be done through a linear transformation $A$ with blocks $(I-H)$ and $(X^TX)^{-1}X^T$

Question 2

assuming all variables have pdfs, you obviously have $P(V=v) = 0$ for any random variable $V$. However as long as you have that the pdf is non-zero $f_V(v) > 0$ you can define conditional pdfs as $$ f(w\mid v) = \dfrac{f_{W,V}(w,v)}{f_V(v)}$$ and compute expectations using $$E[\phi(W)\mid V=v] = \int\phi(w)f(w\mid v)\mathrm{d}w $$ For example think of $f(W\mid V=v)$ as being $W\sim\mathcal{N}(v,\sigma^2)$. You have zero probability, but the pdf changes as the variable $v$ changes.

$\endgroup$
3
  • $\begingroup$ Thanks. So what you're saying is, since $e \sim N(0, \sigma^2(I-H))$, where $H = X(X^TX)^{-1}X^T$, then $P(e = e_0) = 0$ because $e$ is a continuous random variable, correct? $\endgroup$
    – Adrian
    Commented Jan 26, 2021 at 13:58
  • $\begingroup$ $P(e=e_0)$ is indeed 0 but thats not a problem, since the pdf $f_E$ is normal and therefore never 0 $\endgroup$ Commented Jan 26, 2021 at 14:22
  • $\begingroup$ Could you explain what does it intuitively mean to condition on $e = e_0$? i.e., for question 1: "does this mean that we're assuming that $y = y_0$ and $\hat{\beta} = \hat{\beta}^*$ such that $e_0 = y_0 - X\hat{\beta}^*$? That is, we are conditioning on a specific vector of realizations for the responses $y_0$ and a specific vector of OLS coefficients $\hat{\beta}^*$?" $\endgroup$
    – Adrian
    Commented Jan 26, 2021 at 14:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.