Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I'm performing dimension reduction on some data sets and would like to evaluate how has a particular dimension reduction algorithm performed in terms of how much data is lost. If we are given 1000 dimensions, and we reduce it to 2, then how effective is it? I'm trying to figure till what should you do DR such that your results don't go bad? Is there a metric which does this? I'm using PCA.

Edit:

Can I use some distance metric to do the evaulation?

share|improve this question
1  
You don't "lose data" at all, in the sense that all your data points are preserved. You lose some information about how the original points are located relative to each other. To assess losses, you could check how much euclidean distances between the points are underestimated under the reduction done. This is essentially what Stumpy Joe has proposed. –  ttnphns Nov 24 '12 at 6:38
add comment

2 Answers

Let's say you have principal components $v_1$ through $v_n$. Any vector in $n$-space can be translated into the basis of those principal vectors:

$x = x_1v_1 + \dots + x_nv_n $

When you reduce a vector in $n$-space to $d$-space, you are projecting onto the first $d$ principal components and zero-ing out all the rest:

$\hat x = x_1v_1 + \dots + x_dv_d + 0v_{d+1} + \dots + 0v_n$

So if you want to know the error, that would be all the zeroed out parts. I suspect the easiest way to do this is:

given $x$ and precomputed principal components $v_1 \dots v_n$.

$x_1 \dots x_d := $ projection of $x$ onto first $d$ principal components

$\hat x := x_1v_1 + \dots + x_dv_d $

$\|x-\hat x\|^2$ is the squared error of dimensionality reduction

share|improve this answer
add comment

In fact their is a deep lemma about dimension reduction, the Johnson-Lindenstrauss lemma, which asserts that given a set $A = \{a_1,\dots,a_n \}$, a map $f : \mathbb{R}^D \rightarrow \mathbb{R}^d$ is an $\epsilon$-isometry if for every pair $a,a^{'} \in A$ we have $$ ( 1 - \epsilon) || a - a^{'} ||^2 \leq || f(a) - f(a^{'}) ||^2 \leq ( 1 + \epsilon) || a - a^{'} ||^2 $$ and the Johnson-Lindenstrauss lemma asserts that there exists a linear $\epsilon$-isometry whenever $d \geq k \epsilon^{-2} \log ( n )$ where $k$ is an absolute constant. You can realize such a map with an i.i.d. gaussian entries matrix.

Edit : oh sorry i wasnt clear enough about how i think it relates to the question (maybe i am wrong) for dimension reduction (at least in a context where only the distance between points is important like clustering for example) i would calculate how much my mapping (PCA here i think) is changing the distance between points and compare the upper bound of that distorsion to the bound given by JL for example if you go to dimension 2 i would compare it to $\sqrt(\frac{\log(n)}{2})$ if i have n points (of course here 2 is much too low to get a non trivial bound). It gives me a way to assert that my algorithm has the best behavior possible (even if usually i guess you think the other way around setting first an $\epsilon$ and getting a $d$). Another thing the result gives me is a way to construct the best algorithm for dimension reduction which is to use a matrix whose entries are normal random variables as my "projection".

share|improve this answer
    
This is very interesting, thank you. But could you explain how it answers the question? –  whuber Nov 24 '12 at 16:54
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.