I'm in the process of working through Exercises in EM. I am having trouble understanding the first exercise. In particular, I am struggling with deriving the conditional expected value (2) and (3).

In both equations, there appear to be a logistic function. How was this obtained?

In (1), I see how this the log-likelihood belongs to the exponential family, and given $E_i=1$, $E[X_i | \mathcal{Y}] = t + \theta$, but how was the conditional expectation given $E_i = 0$ obtained?

Below is the setup for the first exercise.

The First Exercise

Suppose there are two light bulb survival experiments. In the first, there are $N$ bulbs whose exact lifetimes $y_i$ for $i \in \{1,...,N\}$ are recorded. The lifetimes have an exponential distribution, such that $y_i \sim Exp(\theta).$ In the second experiment, there are $M$ bulbs. After some time (t > 0), a researcher walks into the room and only records how many lightbulbs are still burning out of $M$ bulbs. Thus, the results from the second experiment are right- or -left-censored, and the available data are indicators $E_1, ..., E_M$ for each of the bulbs in the second experiment. If the bulb is still burning, $E_i = 1$, else $E_i = 0$.

Having this data, which is the MLE $\hat\theta$?

Let $X_1, ... , X_M$ be the (unobserved) lifetimes for the second experiment, and let $Z = \sum_{i=1}^ME_i$ be the number of light bulbs still burning. Thus, the observed data from both the experiments combined is $\mathcal{Y} = (Y_1, ..., Y_N, E_1,...,E_M)$ and the unobserserved data is $\mathcal{X} = (X_1, ..., X_M).$

The complete data log-likelihood is

(1) $$ log^c(L(\theta|\mathcal{Y,X})) = -N(log(\theta) + \bar{Y}/\theta) - \sum_{i=1}^M(log(\theta) + X_i/\theta ) $$

which is linear for unobserved $X_i$. But

(2) $$ E[X_i | \mathcal{Y}] = E[X_i | E_i] = \begin{cases} t + \theta & \quad \text{if } E_i = 1\\ \theta - t \frac{e^{-t/\theta}}{1 - e^{-t/\theta}} & \quad \text{if } E_i = 0\\ \end{cases} $$

and therefore the $j$th step consists of replacing $X_i$ in (1) by its expected value (2), using the current numerical parameter value $\theta^{(j-1)}$. The result is

(3) $$ \log(L(\theta)) = -(N + M) log(\theta) - \frac{1}{\theta} [N \bar{Y} + Z ( t + \theta^{(j-1)}) + (M - Z) (\theta^{(j-1)} - t p^{(j-1)})] $$


$$ p^{(j)} = \frac{e^{-t/\theta^{(j)}}}{1 - e^{-t/\theta^{(j)}}} $$

There is more (not shown here) in the paper, but my main concern is how $p^{(j)}$ is obtained.


The event $\{E_i = 0\}$ is the same as $\{X_i < t\}$. Hence \begin{align*} \mathbb{E}(X_i \mid E_i = 0) &= \int_0^t \mathbb{P}(X_i>x\mid E_i = 0) \,\mathrm{d}x \\ &= \int_0^t \mathbb{P}(X_i>x\mid X_i < t) \,\mathrm{d}x \\ &= (1-e^{-t/\theta})^{-1}\int_0^t \mathbb{P}(x < X_i < t) \,\mathrm{d}x \\ &= (1-e^{-t/\theta})^{-1}\int_0^t (e^{-x/\theta} - e^{-t/\theta}) \,\mathrm{d}x \\ &= \theta - t\frac{e^{-t/\theta }}{1-e^{-t/\theta}}. \end{align*}

$\mathbb{E}(X_i\mid E_i=1) = \theta + t$ of course follows from the memoryless property. $(3)$ is obtained from $(1)$ by taking the expectation (under $\theta^{(j-1)}$) of $\sum X_i$ conditional on $Z$ and using $(2)$.


Notationally, we set $Y$ to be the failure time, $Z$ to be the censoring time, and prefer $T$ to represent the minimum of the failure times or the censor times which is what we actually observed. We use a separate indicator vector $C$ to denote whether the event time $T$ corresponds to a censored observation or not.

Exponential RVs are memoryless so for an observation $i$, $E[Y_i|C_i=1] = T_i + \theta$ if $\theta$ is parameterized as a mean ($f_X = \exp (-x/\theta )/\theta$. That comprises the E step for censored observations. The critical assumption here is non-informative censoring meaning that the process $Z$ is independent of $Y$. If an observation $i$ is not censored, then $E[Y_i|C_i=0] = T_i $ since it is fully observed. This is why the EM is trivial when data are fully observed, you can just replace $Y$ with $T$.

For the expression of $p^{(j)}$, If you know the event time $T$ and that the observation is censored, the failure MUST have happened at some time after $T$. The likelihood then comes from a truncated exponential likelihood, which is exactly as written: the full exponential likelihood with 0 density at times before $T$ and scaled by the survivor probability at $T$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.