# What is the $i$th sufficient statistic in the EM algorithm for Gaussian mixture models?

I am reading up on the EM algorithm for Gaussian Mixture Models, and there is consistent reference to the $i$th sufficient statistic. What is this, and why is it relevant to the algorithm?

-
Perhaps you can provide a link to the particular reference you are using. Do you know what a sufficient statistic is in general? What does the letter $i$ index? –  cardinal Aug 7 '11 at 21:51
This was what I figured. There is no use of 'i' anywhere else, which is why I thought that maybe this was something in itself. My reference is this video right now: youtube.com/watch?v=WaKNSBeDLTw –  coffee Aug 7 '11 at 21:56

This question concerns a lecture on EM with an exponential family of distributions. The logarithm of each density $f$ in this family can be expressed (up to a normalization constant) as
$$\log(f_X(x|\mathbf{\theta})) = \eta(x) + \sum_i s_i(x) \theta_i$$
where $\mathbf{\theta} = (\theta_i)$ is a vector of parameters. Apart from influencing the normalization constant (which doesn't matter for inference), parameter $\theta_i$ enters the formula solely through its product with $s_i(x)$, a function of the observation $x$. Thus, everything that can be known about $f$ depends on $x$ only through the vector of values $\mathbf(s) = (s_i(x))$. It is a short step to conclude that the functions $s_i$ generate a set of sufficient statistics for this family. The lecture uses the term "ith sufficient statistic" to refer to $s_i$.
This form of distribution function is one of the simplest possible in any theory or algorithm, like EM, that relies on the log likelihood function, because the log likelihood is a $\theta$-linear combination of the sufficient statistics: the linearity implies the log-likelihood is a differentiable function of the parameters and the partial derivatives can be instantly recognized (they are obtained by summing each $s_i(x)$ over the dataset). Thus it leads to relatively simple formulas and a mathematically tractable analysis.