Matrix dot product in gaussian multivariate distribution

Question

I am having a bit of trouble understanding how the matrix multiplication is carried out in the exponent term of the multivariate gaussian distribution.

I am going to call the covariance matrix C.

In the exponent term for the multivariate gaussian, it looks like -1/2 (x-u)^T C^-1 (x-u)

Lets assume we have x be a 600x2 dataset (2 features), and the covariance matrix be a 2x2 matrix In that case, C^-1 would be a 2x2 matrix, (x-u) would be a 600x2 matrix, thus (x-u)^T would be a 2x600 matrix.

In the matrix product order, (x-u)^T * C^-1 is dotting a 2x600 matrix to a 2x2 matrix, which does not match the dimensions. Similarly, if first doing C^-1 * (x-u)^T, this dots a 2x2 with a 600x2, which also is unfeasible dimensionally. Thus, is the proper way to always compute this to:

First compute (x-u)^T (dot) (x-u) , and then dot that with C^-1 , with C^-1 on the right?

Or am I looking past an important detail?

Answer 1

Usually the equation you've written is given as $$(x - \mu) C^{-1} (x - \mu)^T,$$ with the transpose on the second $(x-\mu)$ term. Written in this way, $x, \mu$ are assumed to be row vectors. As you've written it, $x$ and $\mu$ must be column vectors. Let us call their dimension (the number of features) $m$. If we have $n$ such observations, then the matrix form of $x$ is $$X=\begin{pmatrix} | & & | \\ x_1 & \cdots & x_n \\ | & & |\end{pmatrix}\in \mathbb R^{m \,\times\, n}$$

So, just transpose your matrix $(600, 2) \rightarrow (2, 600)$ and things work out.

The convention given in the first equation is more common because if $x$ is a row vector, then the design matrix of $n$ observations $X$ has dim $(n, m)$, which is how textbooks usually write it.