2
$\begingroup$

I am trying to understand how the Scipy is calculating the score of a sample in the Gaussian Mixture model(log-likelihood).

Below is the equation I got for log-likelihood from the book C.M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.

In my code I am using the following parameters:

gmm = GaussianMixture(n_components=2, covariances_type = 'diag',random_state=0)

I can run gmm.score(X) to get the log-likelihood of the sample. When I investigated the source code, it was not using the determinant or inverse of the covariance. Instead, it was using Cholesky precision matrix.

def _estimate_log_prob(self, X):
   return _estimate_log_gaussian_prob(
   X, self.means_, self.precisions_cholesky_, self.covariance_type)



def _estimate_log_gaussian_prob(X, means, precisions_chol, covariance_type):
    log_det = _compute_log_det_cholesky(
    precisions_chol, covariance_type, n_features)

[...]

    elif covariance_type == 'diag':
       precisions = precisions_chol ** 2
       log_prob = (np.sum((means ** 2 * precisions), 1) -
                  2. * np.dot(X, (means * precisions).T) +
                np.dot(X ** 2, precisions.T))

[...]
    return -.5 * (n_features * np.log(2 * np.pi) + log_prob) + log_det


def _compute_log_det_cholesky(matrix_chol, covariance_type, n_features):
[...]
    elif covariance_type == 'diag':
        log_det_chol = (np.sum(np.log(matrix_chol), axis=1))
[...]
    return log_det_chol

This post explained the mathematics behind it, which is great. But I am confused about the following:

  1. If = np.sum(np.log(matrix_chol), would = np.prod(matrix_chol)?
  2. How is = (np.sum((means ** 2 * precisions_chol ** 2), 1) - 2. * np.dot(X, (means * precisions_chol ** 2).T) + np.dot(X ** 2, precisions_chol ** 2.T))

I would appreciate any answers or feedback from anyone.

Have a great day!

$\endgroup$
4
$\begingroup$
  1. The determinant of a diagonal matrix is the product of the diagonal elements. You can prove this using cofactor expansion along the first column and applying the induction hypothesis.

  2. The second equation just applies the distributive property of multiplication.

$\endgroup$
9
  • $\begingroup$ I appreciate your prompt reply. self.precisions_cholesky_ is a n x 1 matrix, where the n is the number of rows of X. Is that the diagonal elements of the matrix but in a column vector? Thanks for that second answer. I will use that to get likelihood. $\endgroup$ Mar 25 at 3:22
  • $\begingroup$ Presumably. The definition of a diagonal $n\times n$ matrix implies that only $n$ elements that might be nonzero. $\endgroup$
    – Sycorax
    Mar 25 at 3:23
  • $\begingroup$ Say that we have a n x m matrix. Shouldnt the Cov matrix be m x m? That means Cholesky's precision matrix should be m x m correct? $\endgroup$ Mar 25 at 3:24
  • $\begingroup$ Some texts/software uses the convention that rows are observations and columns are features; some uses the opposite convention. I don't think your question is well-formed enough to be answerable. $\endgroup$
    – Sycorax
    Mar 25 at 3:28
  • $\begingroup$ Sorry about that. In my text above, m is the number of features. $\endgroup$ Mar 25 at 3:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.