The question ask to build classifiers to label images of handwritten digits. Each image is $8$ by $8$ pixels and is represented as a vector of dimension $64$ by listing all the pixel values in raster scan order. The images are grayscale and the pixel values are between $0$ and $1$. The labels $t$ are $0$, $1$, $2$, ..., $9$ corresponding to which character was written in the image. There are $700$ training cases and $400$ test cases for each digit.
Using maximum likelihood, fit a set of 10 class-conditional Gaussians with a separate, full covariance matrix for each class. Remember that the conditional multivariate Gaussian probability density is given by, $$p(x |t = k) = (2\pi)^{-D/2}|\Sigma_k|^{-1/2}\text{exp}\bigg\{-\frac{1}{2}(\text{x}-\pmb{\mu}_k)^{\text{T}}\Sigma_k^{-1}(\text{x}-\pmb{\mu}_k)\bigg\}$$ where $\pmb{\mu}_k \in \mathbb{R}^{1\times D}$, $\Sigma_k \in \mathbb{R}^{D\times D}$ and positive-definite. Assume that $p(t=k)=\frac{1}{10}$.

Given that $D=64$ and $N=7000$, and $\text{x}\in\mathbb{R}^{N\times D}$ which $\text{x}$ is a $7000 \times 64$ matrix.

$\textbf{(1)}$ First compute parameter $\mu_{kj}$ and $\Sigma_K$ for each $k\in{0,..,9}$ and $j\in\{1,...,64\}$. Note: To ensure numerical stability you may have to add a small multiple of the identity to each covariance matrix. For this question should add 0.01I to each covariance matrix.
After use numpy in python I find out the $\mu_{kj}$ which is a $1 \times 64$ vector, and $\Sigma_K$ is a $64 \times 64$ matrix, for each $k\in{0,..,9}$ and $j\in\{1,...,64\}$.

$\textbf{(2)}$ Compute $\log(p(x|t))$.
Notice that $$\log(p(x|t)) = -\frac{D}{2}\log(2\pi) - \frac{1}{2}\log(|\Sigma_k^{-1}|) -\frac{1}{2}(\text{x}-\pmb{\mu}_k)^{\text{T}}\Sigma_k^{-1}(\text{x}-\pmb{\mu}_k)$$

But the problem is that $(\text{x}-\pmb{\mu}_k)^{\text{T}}$ is a $64 \times 7000$ matrix in python
and $\Sigma_k^{-1}$ is a $64 \times 64$ matrix in python
and $(\text{x}-\pmb{\mu}_k)$ is a $7000 \times 64$ matrix in python
So, that is impossible to compute
$$(\text{x}-\pmb{\mu}_k)^{\text{T}}\Sigma_k^{-1}(\text{x}-\pmb{\mu}_k)$$ So, I am stuck on computing $\log(p(x|t))$
So, my question is that are there any other ways to compute $\log(p(x|t))$ ?


1 Answer 1


There are closed-form solutions to the MLEs in this case since the membership probabilities are given (sample mean and sample variance-covariance matrix) for observations within each class.

However, your set-up is flawed. Let $\boldsymbol{x}_{ik} \in \mathbb{R}^{64}$ denote the $i$th vectorized 8x8 image for the $k$th label. The last term of the full log-likelihood should be $\sum_{k=0}^9 \sum_{i=1}^{700} \left(\boldsymbol{x}_{ik} - \boldsymbol{\mu}_k\right)^{\prime} \Sigma_k^{-1} \left(\boldsymbol{x}_{ik} - \boldsymbol{\mu}_k\right)$.

You can group all the terms together as well. For instance, let \begin{eqnarray*} \boldsymbol{X}_k = \begin{pmatrix} \boldsymbol{x}_{1,k}^{\prime} \\ \ldots \\ \boldsymbol{x}_{700,k}^{\prime} \end{pmatrix}. \end{eqnarray*} Then we may write the last term of the full log-likelihood as $\sum_{k=0}^9 \left(\mbox{vec} \left(\boldsymbol{X}_k^{\prime}\right) - \boldsymbol{1}_{700} \otimes \boldsymbol{\mu}_k\right)^{\prime} \Sigma_k^{-1} \left(\mbox{vec} \left(\boldsymbol{X}_k^{\prime}\right) - \boldsymbol{1}_{700} \otimes \boldsymbol{\mu}_k\right)$, where $\boldsymbol{1}_{700}$ is a vector of seven hundred 1s.

Finally, the second term is also incorrect in your full log-likelihood. It should be $-350\sum_{k=0}^9 \ln \left|\Sigma_k\right| $


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.