2
$\begingroup$

Let $\Sigma$ be a covariance matrix and let

$$x^T \Sigma^{-1} x = \|Ax\|_2^2.$$ What is the interpretation of matrix $A$?

I tried solving for $A$ with an eigenvalue decomposition of $\Sigma$ as follows,

$$x^T \Sigma^{-1} x = x^T (Q \Lambda Q^T)^{-1}x = x^T Q\Lambda^{-1}Q^{-1} x = x^TA^TAx,$$ and this is where I get stuck. I am unable to figure out a closed form solution to $A$.

$\endgroup$
  • $\begingroup$ Can you figure out the square root of $\Lambda^{-1}$? Do you know what $Q^{-1}$ inverse is? $\endgroup$ – Brian Borchers Mar 3 '15 at 15:16
  • $\begingroup$ Yep, just figured it out to be $\frac{1}{\sqrt{\lambda}}$ where $\lambda$ is the eigenvalues of the covariance matrix $\endgroup$ – user1943079 Mar 3 '15 at 19:03
  • 1
    $\begingroup$ This is a good forum for this question. $\endgroup$ – user603 Mar 3 '15 at 22:55
  • 1
    $\begingroup$ I would still like to get an intuitive understanding of what matrix A means $\endgroup$ – user1943079 Mar 3 '15 at 23:02
  • 1
    $\begingroup$ OK, if it's about interpretation of A (rather than getting a formula for A), then I agree with @user603 that it belongs here. Your original formulation was about "significance of A" and that was not very clear. $\endgroup$ – amoeba Mar 3 '15 at 23:45
2
$\begingroup$

$x^T \Sigma^{-1} x$ is a quadratic form, closely related to the notion of Mahalanobis distance. It's kind of a generalization of standard Euclidean distance in case when not all directions are equal. This is useful when the variation along one direction is much bigger that along the other one, so small distance along the later one is quite significant.

One can decompose $\Sigma^{-1}$ into $Q^T \Lambda Q = Q^T \Lambda^{1/2} \Lambda^{1/2} Q = (\Lambda^{1/2} Q)^T (\Lambda^{1/2} Q)$ (where $\Lambda$ is diagonal matrix of inverted eigenvalues of $\Sigma$, and $Q$ is orthogonal), so when applied to vectors,

$$ x^T \Sigma^{-1} x = x^T (\Lambda^{1/2} Q)^T (\Lambda^{1/2} Q) x = (\Lambda^{1/2} Q x)^T (\Lambda^{1/2} Q x) = \|\Lambda^{1/2} Q x\|^2 $$

What this formula says is that this quadratic form is equivalent to the standard Euclidean distance in new, morphed space. Namely, we apply the transformation $\Lambda^{1/2} Q$ to transform vector $x$, and then take the norm.

What does this transformation do to our $x$? Well, it consists of 2 "sub-transformation": the first one is a multiplication by $Q$, which just rotates the vector (this is what orthogonal matrices do), preserving its norm; and the second one is a multiplication by $\Lambda^{1/2}$ which stretches vectors along each axes in proportion to the square roots of diagonal elements of $\Lambda$.

enter image description here

The picture above shows what will happen with your 2D datapoints if you apply this transformation (for some $\Sigma$). You can think of Mahalanobis distance as of distance, which implicitly maps your vectors into a new space, measures its norm there and returns back to you.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.