0
$\begingroup$

I have two vectors $v_1$, $v_2$. Each can have arbitrary values and I measure their correlation. $v_1$ and $v_2$ can also be different in length and I zero-pad the shorter to macht the longer vector length. Shift invariance of correlation allows to find the the shorter within the longer vector.

Sometimes either one can have all attributes set to 0. If $v_1 = [1,2,3]$ and $v_2 = [0,0,0]$, correlation returns NAN. Mathematically it makes sense, but not in my context. $[0,0,0]$ is a valid entry and I still want to detect the similarity of $v_1$ and $v_2$. How can I capture the correlation or similarity when $v_1$ or $v_2$ is 0?

$\endgroup$
4
  • $\begingroup$ I am surprised that a correlation calculation returns zero, because the correlation in that case is actually undefined. Regardless, your question seems to be saying (1) "I want to use correlation to assess similarity," (2) "But I disagree that correlation is a correct way to assess similarity." What, then, do you expect in the form of an answer? $\endgroup$
    – whuber
    Sep 10, 2014 at 22:09
  • $\begingroup$ There are several continuous data measures. What are your goals/criteria? $\endgroup$
    – dimitriy
    Sep 10, 2014 at 22:25
  • $\begingroup$ Sorry, it returns NAN of course because of the STD division which, in that case is 0. Regarding you reasoning, correlation is a good method for my problem, but I realized that a full zero vector I have need to extend this because it is not treated the way I hoped for. $\endgroup$
    – El Dude
    Sep 10, 2014 at 22:26
  • $\begingroup$ I like correlation because it is shift invariant, which is a pretty versatile feature I would say. $\endgroup$
    – El Dude
    Sep 10, 2014 at 22:27

1 Answer 1

0
$\begingroup$

The purpose of a measure of similarity is to compare two lists of numbers (e.g., vectors), and compute a single number which evaluates their similarity, or distance. The shorter the distance, the greater the level of similarity. Robust distance measures should be generally invariant to scaling.

Pearson's correlation is in fact just one of these similarity or distance measures. For example, correlation distance is defined as: $$1 - corr(v1,v2)$$ That is, the higher the correlation, the smaller the distance, and the greater the level of similarity. However, this measure does not suits you well here since zero vector will result in undefined correlation. So I suggest you to try other distance measures, such as:

  1. Spearman's rank correlation distance: $$ d_{spearman} = 1 - corr(rank(v1, v2))$$

  2. Manhattan distance: $$d_{manhattan} = \sum|v_{1,i} - v{2,i}|$$

  3. Hamming distance, which measures the percentage of coordinats that differ, or the minmum number of substitutes required to change one vector into another.

  4. Chebchev or chessboard distance, which is the extreme case of the general Minkowski metric when p is approaching infinity:

$$ d_{chessboard} = max(|v1 - v2|)$$

  1. And of course the simple Euclidean distance.

In fact, there are too many distance measures, and which one to choose is more depending on your application. Another useful source of inspiration may be this link. That discussion is more on distance analysis of financial time-series, but the general methodology are transferable.

$\endgroup$
1
  • $\begingroup$ Thanks for the suggestions. I edited the main question regarding variable length of the vectors. I'm looking into your links. Thanks! :) $\endgroup$
    – El Dude
    Sep 11, 2014 at 17:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.