I am trying to understand how the Scipy is calculating the score of a sample in the Gaussian Mixture model(log-likelihood).

Below is the equation I got for log-likelihood from the book C.M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.

In my code I am using the following parameters:

gmm = GaussianMixture(n_components=2, covariances_type = 'diag',random_state=0)

I can run gmm.score(X) to get the log-likelihood of the sample. When I investigated the source code, it was not using the determinant or inverse of the covariance. Instead, it was using Cholesky precision matrix.

def _estimate_log_prob(self, X):
   return _estimate_log_gaussian_prob(
   X, self.means_, self.precisions_cholesky_, self.covariance_type)

def _estimate_log_gaussian_prob(X, means, precisions_chol, covariance_type):
    log_det = _compute_log_det_cholesky(
    precisions_chol, covariance_type, n_features)


    elif covariance_type == 'diag':
       precisions = precisions_chol ** 2
       log_prob = (np.sum((means ** 2 * precisions), 1) -
                  2. * np.dot(X, (means * precisions).T) +
                np.dot(X ** 2, precisions.T))

    return -.5 * (n_features * np.log(2 * np.pi) + log_prob) + log_det

def _compute_log_det_cholesky(matrix_chol, covariance_type, n_features):
    elif covariance_type == 'diag':
        log_det_chol = (np.sum(np.log(matrix_chol), axis=1))
    return log_det_chol

This post explained the mathematics behind it, which is great. But I am confused about the following:

  1. If = np.sum(np.log(matrix_chol), would = np.prod(matrix_chol)?
  2. How is = (np.sum((means ** 2 * precisions_chol ** 2), 1) - 2. * np.dot(X, (means * precisions_chol ** 2).T) + np.dot(X ** 2, precisions_chol ** 2.T))

I would appreciate any answers or feedback from anyone.

Have a great day!


1 Answer 1

  1. The determinant of a diagonal matrix is the product of the diagonal elements. You can prove this using cofactor expansion along the first column and applying the induction hypothesis.

  2. The second equation just applies the distributive property of multiplication.

  • $\begingroup$ I appreciate your prompt reply. self.precisions_cholesky_ is a n x 1 matrix, where the n is the number of rows of X. Is that the diagonal elements of the matrix but in a column vector? Thanks for that second answer. I will use that to get likelihood. $\endgroup$ Mar 25, 2021 at 3:22
  • $\begingroup$ Presumably. The definition of a diagonal $n\times n$ matrix implies that only $n$ elements that might be nonzero. $\endgroup$
    – Sycorax
    Mar 25, 2021 at 3:23
  • $\begingroup$ Say that we have a n x m matrix. Shouldnt the Cov matrix be m x m? That means Cholesky's precision matrix should be m x m correct? $\endgroup$ Mar 25, 2021 at 3:24
  • $\begingroup$ Some texts/software uses the convention that rows are observations and columns are features; some uses the opposite convention. I don't think your question is well-formed enough to be answerable. $\endgroup$
    – Sycorax
    Mar 25, 2021 at 3:28
  • $\begingroup$ Sorry about that. In my text above, m is the number of features. $\endgroup$ Mar 25, 2021 at 3:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.