0
$\begingroup$

First off, a little disclaimer: I am basing the question on my own interpretation of the problem. It's well possible that I am mistaken.

Setup: We are given: feature vectors $x_1,\ldots, x_k$ with dimension $N$ such that $k \geq N$ and $\sum_{i=1}^k x_i=0$; $N$ orthonormal basis vectors $v_i$ of dimension $N$. We define approximated vectors $x'$ such that $x'_j = \sum_{i}^d v_i^T x_j v_i$ where $j = 1,\ldots ,k$ and $0 < d < N$.

Problem: Show that the following two methods are equivalent.

  1. minimize the objective function $\sum_{i}^K \|x_i - x'_i\|^2$
  2. maximize $\sum_{i}^d v_i^T S v_i$ where $S$ is the covariance matrix of the vectors $x$

Question: My understanding is that method 1 minimizes the distance between vector $x$ and its approximation $x'$, and that method 2 selects the features with the highest (co)variance. As I am aware, the eigenvector of the covariance matrix with the highest eigenvalue indicates the axis of highest variance, and that, by implication, a line along that axis would be the Least Squares approximator for the data. Therefore, I see how the methods are equivalent in theory, but I'm having trouble finding a mathematical/formal representation of this intuition based on the given expressions.

$\endgroup$
0
$\begingroup$

at least for the trivial case, where the vectors v are rows of the unit Matrix, we can say that method 1 selects the dimensions of x where the elements have the largest absolute values. additionally assuming that the data was centered (i.e. mean = 0), and considering the definition of variance $ \frac 1n \sum(x - \mu)^2$ we can see that method 2 also selects the dimensions where elements have the largest absolute values

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.