# What is the $i$th sufficient statistic in the EM algorithm for Gaussian mixture models?

I am reading up on the EM algorithm for Gaussian Mixture Models, and there is consistent reference to the $i$th sufficient statistic. What is this, and why is it relevant to the algorithm?

-
Perhaps you can provide a link to the particular reference you are using. Do you know what a sufficient statistic is in general? What does the letter $i$ index? –  cardinal Aug 7 '11 at 21:51
This was what I figured. There is no use of 'i' anywhere else, which is why I thought that maybe this was something in itself. My reference is this video right now: youtube.com/watch?v=WaKNSBeDLTw –  coffee Aug 7 '11 at 21:56

This question concerns a lecture on EM with an exponential family of distributions. The logarithm of each density $f$ in this family can be expressed (up to a normalization constant) as

$$\log(f_X(x|\mathbf{\theta})) = \eta(x) + \sum_i s_i(x) \theta_i$$

where $\mathbf{\theta} = (\theta_i)$ is a vector of parameters. Apart from influencing the normalization constant (which doesn't matter for inference), parameter $\theta_i$ enters the formula solely through its product with $s_i(x)$, a function of the observation $x$. Thus, everything that can be known about $f$ depends on $x$ only through the vector of values $\mathbf(s) = (s_i(x))$. It is a short step to conclude that the functions $s_i$ generate a set of sufficient statistics for this family. The lecture uses the term "ith sufficient statistic" to refer to $s_i$.

This form of distribution function is one of the simplest possible in any theory or algorithm, like EM, that relies on the log likelihood function, because the log likelihood is a $\theta$-linear combination of the sufficient statistics: the linearity implies the log-likelihood is a differentiable function of the parameters and the partial derivatives can be instantly recognized (they are obtained by summing each $s_i(x)$ over the dataset). Thus it leads to relatively simple formulas and a mathematically tractable analysis.

-
Your description is good answer for the general scenario. Just to make the discussion a bit clearer in context of EM algorithm (as OP's original question), I would also add: because the EM algorithm replaces the missing observations in the complete data by their conditional expectations, in the context of exponential family, this is equivalent to replacing the missing observations in the sufficient statistics by their conditional expectations. –  suncoolsu Aug 15 '11 at 18:26