0
$\begingroup$

I am having a bit of trouble understanding how the matrix multiplication is carried out in the exponent term of the multivariate gaussian distribution.

I am going to call the covariance matrix C.

In the exponent term for the multivariate gaussian, it looks like -1/2 (x-u)^T C^-1 (x-u)

Lets assume we have x be a 600x2 dataset (2 features), and the covariance matrix be a 2x2 matrix In that case, C^-1 would be a 2x2 matrix, (x-u) would be a 600x2 matrix, thus (x-u)^T would be a 2x600 matrix.

In the matrix product order, (x-u)^T * C^-1 is dotting a 2x600 matrix to a 2x2 matrix, which does not match the dimensions. Similarly, if first doing C^-1 * (x-u)^T, this dots a 2x2 with a 600x2, which also is unfeasible dimensionally. Thus, is the proper way to always compute this to:

First compute (x-u)^T (dot) (x-u) , and then dot that with C^-1 , with C^-1 on the right?

Or am I looking past an important detail?

$\endgroup$
2
  • $\begingroup$ Wikipedia is very clear that it considers $x$ to be a "k-dimensional column vector." You seem to be using "x" both for the model matrix "$X$" and for such individual observations $x$. Case matters! Doesn't that settle the issue? $\endgroup$ – whuber Sep 13 '17 at 21:28
  • $\begingroup$ @whuber absolutely, it seems like my mistake was 100% how i interpreted the x matrix! $\endgroup$ – hundred_dolla_tea Sep 13 '17 at 21:43
0
$\begingroup$

Usually the equation you've written is given as $$(x - \mu) C^{-1} (x - \mu)^T,$$ with the transpose on the second $(x-\mu)$ term. Written in this way, $x, \mu$ are assumed to be row vectors. As you've written it, $x$ and $\mu$ must be column vectors. Let us call their dimension (the number of features) $m$. If we have $n$ such observations, then the matrix form of $x$ is $$X=\begin{pmatrix} | & & | \\ x_1 & \cdots & x_n \\ | & & |\end{pmatrix}\in \mathbb R^{m \,\times\, n}$$

So, just transpose your matrix $(600, 2) \rightarrow (2, 600)$ and things work out.

The convention given in the first equation is more common because if $x$ is a row vector, then the design matrix of $n$ observations $X$ has dim $(n, m)$, which is how textbooks usually write it.

$\endgroup$
2
  • $\begingroup$ thank you for taking the time to help me out. I would like to point out that every literature I have found on the multivariate gaussian uses the formula that I originally posted, from wikipedia, to stanford.edu links, etc. However, the math naturally makes much more sense if I could move around the transpose, I am assuming that my issue is the way my training data is structured (in my case, it is a matrix with each row representing a data point, consisting of 2 features (so two columns) ) $\endgroup$ – hundred_dolla_tea Sep 13 '17 at 20:55
  • $\begingroup$ That's fine-- we apparently use a different set of texts. Either way, my original answer still stands. Transpose your X matrix so that the feature vectors are stacked column-wise, and it works out. $\endgroup$ – the higgs broson Sep 13 '17 at 20:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.