0
$\begingroup$

I want to correlate one sample to a set of classes' centroids (i.e. for each class, the sample composed by the median value of each feature in the set of samples of the class) to understand which class the sample is most correlated with.

However, I noticed that the sample I want to correlate is a sparse array, i.e. most of the entries are equal to $0$.

The classes' centroids are not sparse.

May a correlation coefficient (such as Pearson's r) be a good way to assess the correlation between the sample and the classes' centroids or can it be affected by the sample's sparsity?

Improve this question
$\endgroup$
1
  • $\begingroup$ I want to correlate is a sparse array In what sense you use "sparsity"? is "0" a NA or 0 value? In the latter case, note that 0 for scale (interval, ratio) data is a not a sign of sparsity, it is just such value. We say of 0 as sparsity in binary or count data. But for binaty or count data, it is questionable whether notion of "centroid" (euclidean mean) is applicable at all. $\endgroup$
    – ttnphns
    Commented Nov 19, 2016 at 9:16

1 Answer 1

Reset to default
1
$\begingroup$

Correlations do not work very well if you have a high sparsity.

Recall what a correlation is meant to do: check if the points in one set follow a similar ordering/trend as in the other set. But if most points are "absent" (unranked, tied at the same rank) then this doesn't work well anymore.

As an example, you can use the vectors

$$ X=\{1, 0, 3, 0, 5, \ldots, N-1, 0\}\\ Y=\{0, 2, 0, 4, 0, \ldots, \,\,\,0\phantom{-1}, N\} $$

Intuitively, we would want these to be correlated, despite the missing values.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.