10
$\begingroup$

In a problem set I proved this "lemma," whose result is not intuitive to me. $Z$ is a standard normal distribution in a censored model.

Formally, $Z^* \sim Norm(0, \sigma^2)$, and $Z = max(Z^*, c)$. Then, \begin{align} E[Z|Z>c] &= \int_c^\infty z_i \phi({z_i})\mathrm{d}z_i \\ &= \frac{1}{\sqrt{2\pi}}\int_c^\infty z_i \exp\!\bigg(\frac{-1}{2}z_i^2\bigg)\mathrm{d}z_i \\ &= \frac{1}{\sqrt{2\pi}} \exp\!\bigg(\frac{-1}{2}c^2\bigg) \quad\quad\quad\quad\text{ (Integration by substitution)}\\ &= \phi(c) \end{align} So there is some sort of connection between the expectation formula over a truncated domain and the density at the point of truncation $(c)$. Could anyone explain the intuition behind this?

Improve this question
$\endgroup$
7
  • 2
    $\begingroup$ That it turns out that way is a consequence of the fact that the $z$ term is the negative of the derivative of the term in the exponent; it's one of many neat results for the standard normal but it doesn't necessarily have intuition behind it. On the other hand it wouldn't surprise me at all if one of the clever people here could come up with some kind of intuition for it. $\endgroup$
    – Glen_b
    Feb 12, 2014 at 21:29
  • $\begingroup$ @Glen_b What you are saying is that $\int_c^\infty \left(-\frac{d}{dz}\log(f(z))\right) f(z) dz = -\int_c^\infty f^\prime(z)dz = f(c)$ where $f$ is the PDF of any continuous distribution $F.$ $\endgroup$
    – whuber ♦
    Feb 12, 2014 at 22:10
  • $\begingroup$ @whuber That's certainly the case, and it's worth emphasizing that result, since it's directly relevant to the result in the question, but actually in my comment I was referring specifically to the case where the first of those terms is $z$ (since the term "the expectation formula" was in the question, I took it to be about $E(Z|Z>c)$, which is specific to the normal. $\endgroup$
    – Glen_b
    Feb 12, 2014 at 22:16
  • $\begingroup$ (at least up to the obvious multiplicative constant, about that conditional expectation). However, $E(g(Z)|Z>c)$ for that particular $g=-\frac{d}{dz}log f$ is probably worth discussing in an answer. $\endgroup$
    – Glen_b
    Feb 12, 2014 at 22:21
  • 1
    $\begingroup$ Your latest edit asks for a proof (or intuitive explanation) of an incorrect statement. The conditional density of $Z \sim N(0,1)$ conditioned on $Z > c$ is $\frac{\phi(z)}{1-\Phi(c)}\mathbf 1_{\{z\colon z > c\}}$ and the conditional expected value is thus $$E[Z\mid Z > c] = \int_c^{\infty}z\frac{\phi(z)}{1-\Phi(c)}\,\mathrm dz = \frac{1}{1-\Phi(c)}\int_c^{\infty}z\phi(z)\,\mathrm dz$$ and not what you have in your revised title. $\endgroup$
    – Dilip Sarwate
    Feb 13, 2014 at 4:52

1 Answer 1

Reset to default
6
$\begingroup$

Would the Fundamental Theorem of Calculus work for you as intuition?

Let $\phi(x)$ denote the density function $\frac{1}{\sqrt{2\pi}}e^{-x^2/2}$ of a standard normal random variable. Then, the derivative is $\frac{\mathrm d}{\mathrm dx}\phi(x) = -x\phi(x)$. The Fundamental Theorem of Calculus then gives us that $$\phi(x) = \int_{-\infty}^x -t\phi(t)\,\mathrm dt = \int_{-x}^\infty u\phi(u)\,\mathrm du = \int_x^\infty u\phi(u)\,\mathrm du$$ where the second integral is obtained on substituting $u = -t$ and using the fact that $\phi(-u) = \phi(u)$ and the third upon noting that $\phi(-x) = \phi(x)$. Alternatively, write the second integral as the integral from $-x$ to $+x$ plus the integral from $+x$ to $\infty$, and note that integrating an odd function from $-x$ to $+x$ results in $0$.

Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.