Exercises in EM with censored data

Question

I'm in the process of working through Exercises in EM. I am having trouble understanding the first exercise. In particular, I am struggling with deriving the conditional expected value (2) and (3).

In both equations, there appear to be a logistic function. How was this obtained?

In (1), I see how this the log-likelihood belongs to the exponential family, and given $E_i=1$, $E[X_i | \mathcal{Y}] = t + \theta$, but how was the conditional expectation given $E_i = 0$ obtained?

Below is the setup for the first exercise.

The First Exercise

Suppose there are two light bulb survival experiments. In the first, there are $N$ bulbs whose exact lifetimes $y_i$ for $i \in \{1,...,N\}$ are recorded. The lifetimes have an exponential distribution, such that $y_i \sim Exp(\theta).$ In the second experiment, there are $M$ bulbs. After some time (t > 0), a researcher walks into the room and only records how many lightbulbs are still burning out of $M$ bulbs. Thus, the results from the second experiment are right- or -left-censored, and the available data are indicators $E_1, ..., E_M$ for each of the bulbs in the second experiment. If the bulb is still burning, $E_i = 1$, else $E_i = 0$.

Having this data, which is the MLE $\hat\theta$?

Let $X_1, ... , X_M$ be the (unobserved) lifetimes for the second experiment, and let $Z = \sum_{i=1}^ME_i$ be the number of light bulbs still burning. Thus, the observed data from both the experiments combined is $\mathcal{Y} = (Y_1, ..., Y_N, E_1,...,E_M)$ and the unobserserved data is $\mathcal{X} = (X_1, ..., X_M).$

The complete data log-likelihood is

(1) $$ log^c(L(\theta|\mathcal{Y,X})) = -N(log(\theta) + \bar{Y}/\theta) - \sum_{i=1}^M(log(\theta) + X_i/\theta ) $$

which is linear for unobserved $X_i$. But

(2) $$ E[X_i | \mathcal{Y}] = E[X_i | E_i] = \begin{cases} t + \theta & \quad \text{if } E_i = 1\\ \theta - t \frac{e^{-t/\theta}}{1 - e^{-t/\theta}} & \quad \text{if } E_i = 0\\ \end{cases} $$

and therefore the $j$th step consists of replacing $X_i$ in (1) by its expected value (2), using the current numerical parameter value $\theta^{(j-1)}$. The result is

(3) $$ \log(L(\theta)) = -(N + M) log(\theta) - \frac{1}{\theta} [N \bar{Y} + Z ( t + \theta^{(j-1)}) + (M - Z) (\theta^{(j-1)} - t p^{(j-1)})] $$

where

$$ p^{(j)} = \frac{e^{-t/\theta^{(j)}}}{1 - e^{-t/\theta^{(j)}}} $$

There is more (not shown here) in the paper, but my main concern is how $p^{(j)}$ is obtained.

Answer 1

The event $\{E_i = 0\}$ is the same as $\{X_i < t\}$. Hence \begin{align*} \mathbb{E}(X_i \mid E_i = 0) &= \int_0^t \mathbb{P}(X_i>x\mid E_i = 0) \,\mathrm{d}x \\ &= \int_0^t \mathbb{P}(X_i>x\mid X_i < t) \,\mathrm{d}x \\ &= (1-e^{-t/\theta})^{-1}\int_0^t \mathbb{P}(x < X_i < t) \,\mathrm{d}x \\ &= (1-e^{-t/\theta})^{-1}\int_0^t (e^{-x/\theta} - e^{-t/\theta}) \,\mathrm{d}x \\ &= \theta - t\frac{e^{-t/\theta }}{1-e^{-t/\theta}}. \end{align*}

$\mathbb{E}(X_i\mid E_i=1) = \theta + t$ of course follows from the memoryless property. $(3)$ is obtained from $(1)$ by taking the expectation (under $\theta^{(j-1)}$) of $\sum X_i$ conditional on $Z$ and using $(2)$.

Answer 2

Notationally, we set $Y$ to be the failure time, $Z$ to be the censoring time, and prefer $T$ to represent the minimum of the failure times or the censor times which is what we actually observed. We use a separate indicator vector $C$ to denote whether the event time $T$ corresponds to a censored observation or not.

Exponential RVs are memoryless so for an observation $i$, $E[Y_i|C_i=1] = T_i + \theta$ if $\theta$ is parameterized as a mean ($f_X = \exp (-x/\theta )/\theta$. That comprises the E step for censored observations. The critical assumption here is non-informative censoring meaning that the process $Z$ is independent of $Y$. If an observation $i$ is not censored, then $E[Y_i|C_i=0] = T_i $ since it is fully observed. This is why the EM is trivial when data are fully observed, you can just replace $Y$ with $T$.

For the expression of $p^{(j)}$, If you know the event time $T$ and that the observation is censored, the failure MUST have happened at some time after $T$. The likelihood then comes from a truncated exponential likelihood, which is exactly as written: the full exponential likelihood with 0 density at times before $T$ and scaled by the survivor probability at $T$.