5
$\begingroup$

I'm in the process of working through Exercises in EM. I am having trouble understanding the first exercise. In particular, I am struggling with deriving the conditional expected value (2) and (3).

In both equations, there appear to be a logistic function. How was this obtained?

In (1), I see how this the log-likelihood belongs to the exponential family, and given $E_i=1$, $E[X_i | \mathcal{Y}] = t + \theta$, but how was the conditional expectation given $E_i = 0$ obtained?

Below is the setup for the first exercise.

The First Exercise

Suppose there are two light bulb survival experiments. In the first, there are $N$ bulbs whose exact lifetimes $y_i$ for $i \in \{1,...,N\}$ are recorded. The lifetimes have an exponential distribution, such that $y_i \sim Exp(\theta).$ In the second experiment, there are $M$ bulbs. After some time (t > 0), a researcher walks into the room and only records how many lightbulbs are still burning out of $M$ bulbs. Thus, the results from the second experiment are right- or -left-censored, and the available data are indicators $E_1, ..., E_M$ for each of the bulbs in the second experiment. If the bulb is still burning, $E_i = 1$, else $E_i = 0$.

Having this data, which is the MLE $\hat\theta$?

Let $X_1, ... , X_M$ be the (unobserved) lifetimes for the second experiment, and let $Z = \sum_{i=1}^ME_i$ be the number of light bulbs still burning. Thus, the observed data from both the experiments combined is $\mathcal{Y} = (Y_1, ..., Y_N, E_1,...,E_M)$ and the unobserserved data is $\mathcal{X} = (X_1, ..., X_M).$

The complete data log-likelihood is

(1) $$ log^c(L(\theta|\mathcal{Y,X})) = -N(log(\theta) + \bar{Y}/\theta) - \sum_{i=1}^M(log(\theta) + X_i/\theta ) $$

which is linear for unobserved $X_i$. But

(2) $$ E[X_i | \mathcal{Y}] = E[X_i | E_i] = \begin{cases} t + \theta & \quad \text{if } E_i = 1\\ \theta - t \frac{e^{-t/\theta}}{1 - e^{-t/\theta}} & \quad \text{if } E_i = 0\\ \end{cases} $$

and therefore the $j$th step consists of replacing $X_i$ in (1) by its expected value (2), using the current numerical parameter value $\theta^{(j-1)}$. The result is

(3) $$ \log(L(\theta)) = -(N + M) log(\theta) - \frac{1}{\theta} [N \bar{Y} + Z ( t + \theta^{(j-1)}) + (M - Z) (\theta^{(j-1)} - t p^{(j-1)})] $$

where

$$ p^{(j)} = \frac{e^{-t/\theta^{(j)}}}{1 - e^{-t/\theta^{(j)}}} $$

There is more (not shown here) in the paper, but my main concern is how $p^{(j)}$ is obtained.

$\endgroup$
5
  • $\begingroup$ could you explain how the log-likelihood function was obtained? $\endgroup$
    – Richard
    Aug 3, 2022 at 19:36
  • $\begingroup$ @Richard refer to the linked paper for details Exercises in EM $\endgroup$
    – Jon
    Aug 4, 2022 at 20:03
  • $\begingroup$ but there they don't explain how they obtained it. $\endgroup$
    – Richard
    Aug 4, 2022 at 20:33
  • $\begingroup$ @Richard you should post a topic with that question. It wouldn’t belong here, sorry. It’d be a little off topic $\endgroup$
    – Jon
    Aug 5, 2022 at 21:02
  • $\begingroup$ For $t>0$, we directly have $$E\left[X_i\mid E_i=0\right]=E\left[X_i\mid X_i<t\right]=\frac{E\left[X_iI(X_i<t)\right]}{P(X_i<t)}=\frac1{1-e^{-t/\theta}}\int_0^t \frac x{\theta}e^{-x/\theta}\,dx$$ Alternatively, we can use the following to solve for $E\left[X_i\mid E_i=0\right]$: $$E\left[X_i\right]=E\left[X_i\mid E_i=1\right]P(E_i=1)+E\left[X_i\mid E_i=0\right]P(E_i=0)$$ $\endgroup$ Oct 1, 2022 at 15:55

2 Answers 2

2
+150
$\begingroup$

The event $\{E_i = 0\}$ is the same as $\{X_i < t\}$. Hence \begin{align*} \mathbb{E}(X_i \mid E_i = 0) &= \int_0^t \mathbb{P}(X_i>x\mid E_i = 0) \,\mathrm{d}x \\ &= \int_0^t \mathbb{P}(X_i>x\mid X_i < t) \,\mathrm{d}x \\ &= (1-e^{-t/\theta})^{-1}\int_0^t \mathbb{P}(x < X_i < t) \,\mathrm{d}x \\ &= (1-e^{-t/\theta})^{-1}\int_0^t (e^{-x/\theta} - e^{-t/\theta}) \,\mathrm{d}x \\ &= \theta - t\frac{e^{-t/\theta }}{1-e^{-t/\theta}}. \end{align*}

$\mathbb{E}(X_i\mid E_i=1) = \theta + t$ of course follows from the memoryless property. $(3)$ is obtained from $(1)$ by taking the expectation (under $\theta^{(j-1)}$) of $\sum X_i$ conditional on $Z$ and using $(2)$.

$\endgroup$
2
$\begingroup$

Notationally, we set $Y$ to be the failure time, $Z$ to be the censoring time, and prefer $T$ to represent the minimum of the failure times or the censor times which is what we actually observed. We use a separate indicator vector $C$ to denote whether the event time $T$ corresponds to a censored observation or not.

Exponential RVs are memoryless so for an observation $i$, $E[Y_i|C_i=1] = T_i + \theta$ if $\theta$ is parameterized as a mean ($f_X = \exp (-x/\theta )/\theta$. That comprises the E step for censored observations. The critical assumption here is non-informative censoring meaning that the process $Z$ is independent of $Y$. If an observation $i$ is not censored, then $E[Y_i|C_i=0] = T_i $ since it is fully observed. This is why the EM is trivial when data are fully observed, you can just replace $Y$ with $T$.

For the expression of $p^{(j)}$, If you know the event time $T$ and that the observation is censored, the failure MUST have happened at some time after $T$. The likelihood then comes from a truncated exponential likelihood, which is exactly as written: the full exponential likelihood with 0 density at times before $T$ and scaled by the survivor probability at $T$.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.