3
$\begingroup$

Suppose $Y \in \mathbb{R}^n$ and $Z \in \mathbb{R}^n$ are random vectors, where $Y$ follows a $MVN(\mu, \Sigma)$ distribution. Let $X \in \mathbb{R}^{n \times p}$ be a full-rank fixed matrix of constants. Let $E = Y - \hat{Y}$ denote the residuals of performing OLS regression, where one would obtain $E$ by projecting $Y$ onto the orthogonal complement column space of $X$.

By the law of total expectation, does the following hold?

$$E[g(Z,Y)] = E[g(Z,Y)|\mathcal{D}]P(\mathcal{D}) + E[g(Z,Y)|\mathcal{D}^c]P(\mathcal{D}^c)$$

where $g(\cdot, \cdot)$ is a function, and $A \in \mathbb{R}^{n \times n}$ and $b \in \mathbb{R}^n$ are a matrix and vector of constants, respectively, and $\mathcal{D} = \{AY \geq b, E = E_0\}$.

I wasn't sure if it would hold because of the conditioning event $E=E_0$, which I interpret as conditioning on observing a particular vector of residuals $E_0$. Because the residuals $E$ are continuous, I think the probability of observing $E = E_0$ would be 0. Does that imply the following: $$E[g(Z,Y)] \overset{?}{=} 0 + E[g(Z,Y)|\mathcal{D}^c]P(\mathcal{D}^c)$$

$\endgroup$

2 Answers 2

1
$\begingroup$

$\newcommand{\E}{\operatorname{E}}$I think this is something where a more rigorous definition of conditional expectation helps. Let $(\Omega, \mathscr F, P)$ be a probability space with $X : \Omega\to\mathbb R^n$ a random vector. For a sub-$\sigma$-algebra $\mathcal G$ of $\mathscr F$ the conditional expectation of $X$ on $\mathcal G$, denoted $\E[X\mid \mathcal G]$, is an almost surely unique random variable that is $(\mathcal G, \mathbb B)$-measurable and $$ \int_A \E[X \mid \mathcal G] \,\text dP = \int_A X \,\text dP $$ for all $A \in \mathcal G$.

This is for conditioning on a $\sigma$-algebra, not a single event. If we want to condition on a single event $D$ we can take $\mathscr D := \{\emptyset, D, D^c, \Omega\}$ as the $\sigma$-algebra generated by this event. $\E[X \mid \mathscr D]$ is constant on $D$ and $D^c$ otherwise it's not $\mathscr D$-measurable, so $\E[X\mid D] = \E[X \mid \mathscr D](\omega)$ for $\omega\in D$.

If $P(D)=1$ this means $$ \E[X] = \int_D X\,\text dP = \int_D \E[X\mid \mathscr D]\,\text dP = \E[X \mid D] P(D) = \E[X \mid D]. $$ Thus if $P(D)=1$ then the value that $\E[X\mid\mathscr D]$ takes on $D$ is just $\E[X]$. Intuitively this makes sense because $D^c$ is just a measure zero slice of $\Omega$ that we're shaving off and since $\E[X\mid \mathscr D]$ is only defined almost surely, it would be very strange if we got different behavior by this modification on a null set.

If instead $P(D)=0$ then $$ 0 = \int_D X\,\text dP = \E[X \mid D] P(D) $$ and we can define $\E[X \mid D]$ arbitrarily since its values don't actually affect anything.

$\endgroup$
2
  • $\begingroup$ Thanks for such a detailed explanation. In my example, I think the main thing I am not sure about is whether $P(\mathcal{D}) = 0$, where $\mathcal{D} = \{AY \geq b, E = E_0\}$. I think it might be because $E$ is continuous? What do you think? $\endgroup$
    – Adrian
    Jan 26, 2021 at 16:52
  • $\begingroup$ @Adrian $E = (I-H)Y$ where $H$ is the hat matrix so $E$ still has a multivariate Gaussian, just not a full rank one (the support is on the $n-p$ dimension subspace orthogonal to $\text{ColSpace}(X)$). It'll still have a probability of zero of being any particular vector in that space since it is still continuous there (unless $p=n$ and $(I-H)Y = \mathbf 0$) $\endgroup$
    – jld
    Jan 26, 2021 at 16:57
1
$\begingroup$

If $P(\mathcal{D}) = 0$ then $P(\mathcal{D}^c) = 1$, this is because $\{\mathcal{D}, \mathcal{D}^c\}$ are a disjoint set: $\mathcal{D}$ can happen or it cannot. Plugging these values into the relevant expression yields $E(X) = E(X|D^c)$, which will generalise to $E(g(Z,Y)) = E(g(Z,Y)|D^c)$.

In essence, if you know $\mathcal{D}$ did not happen for sure, then your expected value of $X$, $E(X)$ should be adjusted for the fact that $\mathcal{D}$ did not occur.

$\endgroup$
3
  • $\begingroup$ Is my interpretation of the conditioning event $E = E_0$ correct? That is, in my example, is $P(\mathcal{D}) = 0$ where $\mathcal{D} = \{AY \geq b, E = E_0\}$? $\endgroup$
    – Adrian
    Jan 26, 2021 at 16:25
  • $\begingroup$ I agree with you answer in general, but I think my main confusion stems from the specific example I've outlined in this question. $\endgroup$
    – Adrian
    Jan 26, 2021 at 16:33
  • $\begingroup$ With continuous values data we really need to be thinking about probability density rather than probability. Essentially when we do this sums become integrals which takes care of the fact that $P(E = E_0) = 0$ no matter what $E_0$ is. $\endgroup$
    – jcken
    Jan 26, 2021 at 16:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.