# Matrix dot product in gaussian multivariate distribution

I am having a bit of trouble understanding how the matrix multiplication is carried out in the exponent term of the multivariate gaussian distribution.

I am going to call the covariance matrix C.

In the exponent term for the multivariate gaussian, it looks like -1/2 (x-u)^T C^-1 (x-u)

Lets assume we have x be a 600x2 dataset (2 features), and the covariance matrix be a 2x2 matrix In that case, C^-1 would be a 2x2 matrix, (x-u) would be a 600x2 matrix, thus (x-u)^T would be a 2x600 matrix.

In the matrix product order, (x-u)^T * C^-1 is dotting a 2x600 matrix to a 2x2 matrix, which does not match the dimensions. Similarly, if first doing C^-1 * (x-u)^T, this dots a 2x2 with a 600x2, which also is unfeasible dimensionally. Thus, is the proper way to always compute this to:

First compute (x-u)^T (dot) (x-u) , and then dot that with C^-1 , with C^-1 on the right?

Or am I looking past an important detail?

• Wikipedia is very clear that it considers $x$ to be a "k-dimensional column vector." You seem to be using "x" both for the model matrix "$X$" and for such individual observations $x$. Case matters! Doesn't that settle the issue? – whuber Sep 13 '17 at 21:28
• @whuber absolutely, it seems like my mistake was 100% how i interpreted the x matrix! – hundred_dolla_tea Sep 13 '17 at 21:43

## 1 Answer

Usually the equation you've written is given as $$(x - \mu) C^{-1} (x - \mu)^T,$$ with the transpose on the second $(x-\mu)$ term. Written in this way, $x, \mu$ are assumed to be row vectors. As you've written it, $x$ and $\mu$ must be column vectors. Let us call their dimension (the number of features) $m$. If we have $n$ such observations, then the matrix form of $x$ is $$X=\begin{pmatrix} | & & | \\ x_1 & \cdots & x_n \\ | & & |\end{pmatrix}\in \mathbb R^{m \,\times\, n}$$

So, just transpose your matrix $(600, 2) \rightarrow (2, 600)$ and things work out.

The convention given in the first equation is more common because if $x$ is a row vector, then the design matrix of $n$ observations $X$ has dim $(n, m)$, which is how textbooks usually write it.

• thank you for taking the time to help me out. I would like to point out that every literature I have found on the multivariate gaussian uses the formula that I originally posted, from wikipedia, to stanford.edu links, etc. However, the math naturally makes much more sense if I could move around the transpose, I am assuming that my issue is the way my training data is structured (in my case, it is a matrix with each row representing a data point, consisting of 2 features (so two columns) ) – hundred_dolla_tea Sep 13 '17 at 20:55
• That's fine-- we apparently use a different set of texts. Either way, my original answer still stands. Transpose your X matrix so that the feature vectors are stacked column-wise, and it works out. – the higgs broson Sep 13 '17 at 20:57