Finding out the generative log-likelihood $\log(p(x|t))$

Question

The question ask to build classifiers to label images of handwritten digits. Each image is $8$ by $8$ pixels and is represented as a vector of dimension $64$ by listing all the pixel values in raster scan order. The images are grayscale and the pixel values are between $0$ and $1$. The labels $t$ are $0$, $1$, $2$, ..., $9$ corresponding to which character was written in the image. There are $700$ training cases and $400$ test cases for each digit.
Using maximum likelihood, fit a set of 10 class-conditional Gaussians with a separate, full covariance matrix for each class. Remember that the conditional multivariate Gaussian probability density is given by, $$p(x |t = k) = (2\pi)^{-D/2}|\Sigma_k|^{-1/2}\text{exp}\bigg\{-\frac{1}{2}(\text{x}-\pmb{\mu}_k)^{\text{T}}\Sigma_k^{-1}(\text{x}-\pmb{\mu}_k)\bigg\}$$ where $\pmb{\mu}_k \in \mathbb{R}^{1\times D}$, $\Sigma_k \in \mathbb{R}^{D\times D}$ and positive-definite. Assume that $p(t=k)=\frac{1}{10}$.

Given that $D=64$ and $N=7000$, and $\text{x}\in\mathbb{R}^{N\times D}$ which $\text{x}$ is a $7000 \times 64$ matrix.

$\textbf{(1)}$ First compute parameter $\mu_{kj}$ and $\Sigma_K$ for each $k\in{0,..,9}$ and $j\in\{1,...,64\}$. Note: To ensure numerical stability you may have to add a small multiple of the identity to each covariance matrix. For this question should add 0.01I to each covariance matrix.
After use numpy in python I find out the $\mu_{kj}$ which is a $1 \times 64$ vector, and $\Sigma_K$ is a $64 \times 64$ matrix, for each $k\in{0,..,9}$ and $j\in\{1,...,64\}$.

$\textbf{(2)}$ Compute $\log(p(x|t))$.
Notice that $$\log(p(x|t)) = -\frac{D}{2}\log(2\pi) - \frac{1}{2}\log(|\Sigma_k^{-1}|) -\frac{1}{2}(\text{x}-\pmb{\mu}_k)^{\text{T}}\Sigma_k^{-1}(\text{x}-\pmb{\mu}_k)$$

But the problem is that $(\text{x}-\pmb{\mu}_k)^{\text{T}}$ is a $64 \times 7000$ matrix in python
and $\Sigma_k^{-1}$ is a $64 \times 64$ matrix in python
and $(\text{x}-\pmb{\mu}_k)$ is a $7000 \times 64$ matrix in python
So, that is impossible to compute
$$(\text{x}-\pmb{\mu}_k)^{\text{T}}\Sigma_k^{-1}(\text{x}-\pmb{\mu}_k)$$ So, I am stuck on computing $\log(p(x|t))$
So, my question is that are there any other ways to compute $\log(p(x|t))$ ?

Answer 1

There are closed-form solutions to the MLEs in this case since the membership probabilities are given (sample mean and sample variance-covariance matrix) for observations within each class.

However, your set-up is flawed. Let $\boldsymbol{x}_{ik} \in \mathbb{R}^{64}$ denote the $i$th vectorized 8x8 image for the $k$th label. The last term of the full log-likelihood should be $\sum_{k=0}^9 \sum_{i=1}^{700} \left(\boldsymbol{x}_{ik} - \boldsymbol{\mu}_k\right)^{\prime} \Sigma_k^{-1} \left(\boldsymbol{x}_{ik} - \boldsymbol{\mu}_k\right)$.

You can group all the terms together as well. For instance, let \begin{eqnarray*} \boldsymbol{X}_k = \begin{pmatrix} \boldsymbol{x}_{1,k}^{\prime} \\ \ldots \\ \boldsymbol{x}_{700,k}^{\prime} \end{pmatrix}. \end{eqnarray*} Then we may write the last term of the full log-likelihood as $\sum_{k=0}^9 \left(\mbox{vec} \left(\boldsymbol{X}_k^{\prime}\right) - \boldsymbol{1}_{700} \otimes \boldsymbol{\mu}_k\right)^{\prime} \Sigma_k^{-1} \left(\mbox{vec} \left(\boldsymbol{X}_k^{\prime}\right) - \boldsymbol{1}_{700} \otimes \boldsymbol{\mu}_k\right)$, where $\boldsymbol{1}_{700}$ is a vector of seven hundred 1s.

Finally, the second term is also incorrect in your full log-likelihood. It should be $-350\sum_{k=0}^9 \ln \left|\Sigma_k\right| $