1
$\begingroup$

I am reading the derivation of EM algorithm to estimate the maximum likelihood of a multivariate Student-t distribution $\mathcal{T}(\mathbf{x} \vert \pmb{\mu}, \pmb{\Sigma}, \nu)$ in Kevin Murphy's textbook (Machine Learning: a probabilistic perspective, section 11.4.5). I can follow the whole section, but have a concern at the very last step when determining the dof $\nu$.

The E-step for $\nu$ is: \begin{equation} \mathbb{E}\left[L_{G}(\nu) \right] = -N \log \Gamma \left(\frac{\nu}{2} \right) + \frac{N\nu}{2} \log \left(\frac{\nu}{2}\right) + \frac{\nu}{2} \sum_{i=1}^{N} \left( \bar{\ell}^{(t)}_{i} - \bar{z}_{i}^{(t)} \right), \tag{11.77} \label{11.77} \end{equation} where $\Gamma(x)$ is the gamma function.

The M-step is to maximize $\mathbb{E}\left[L_{G}(\nu) \right]$ w.r.t. $\nu$ by solving for the root of the first derivative: \begin{equation} \frac{\partial \mathbb{E}\left[L_{G}(\nu) \right]}{\partial \nu} = -\frac{N}{2} \Psi \left(\frac{\nu}{2} \right) + \frac{N}{2} \log \left(\frac{\nu}{2} \right) + \frac{N}{2} + \frac{1}{2}\sum_{i=1}^{N} \left( \bar{\ell}^{(t)}_{i} - \bar{z}_{i}^{(t)} \right), \tag{11.78} \label{11.78} \end{equation} where $\Psi(x)$ is the digamma function.

As discussed in the textbook, the root of \eqref{11.78} can be found by using some numerical methods for root finding, such as half-interval method. However, I wonder if we can apply Newton-Raphson method for \eqref{11.78} by taking second derivative: \begin{equation} \frac{\partial^2 \mathbb{E}\left[L_{G}(\nu) \right]}{\partial\nu^{2}} = -\frac{N}{4} \Psi^{(2)} \left(\frac{\nu}{2} \right) + \frac{N}{2\nu}, \end{equation} where $\Psi^{(2)}(x)$ is the trigamma function.

Given the second derivative, we can apply the Newton-Raphson method to iteratively find the dof $\nu$ as: \begin{equation} \nu \gets \nu - \left(\frac{\partial \mathbb{E}\left[L_{G}(\nu) \right]}{\partial \nu}\right)/\left(\frac{\partial^2 \mathbb{E}\left[L_{G}(\nu) \right]}{\partial\nu^{2}}\right). \end{equation}

My question is if we can apply Newton-Raphson method to find $\nu$. What are the disadvantages of the Newton-Raphson method preventing us from using in this case? I am aware that Newton-Raphson method requires the second derivative, but in this case, the second derivative can be efficiently calculated. Or, I might be wrong because of the complexity of the digamma calculation.

$\endgroup$
1
$\begingroup$

You're correct, the Newton-Raphson method could be used here as well. In fact, you can treat it as a root-finding method applied to the (11.78).

The trigamma function does not introduce challenge either. Surely it's not available in a closed form, but so does digamma, and there're efficient numerical procedures to compute both of them, in fact scipy has a generic scipy.special.polygamma function that covers both of them.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.