0
$\begingroup$

I calculated the Cosine distances for binary data and got the relations between different variables.

I need to cluster them. I tried passing the cosine matrix directly to the (clustering) function but got an error. So, I tried passing the cosine matrix as a parameter to the dist function (dist computes Euclidean distances between all rows in a matrix.)

d <- dist(cosine(matrix1))

and then plotted the result with hc<- hclust(d). I am getting good output but want to check whether I am doing anything wrong.

Does it makes sense to pass cosine distances to the dist function?

$\endgroup$
  • 1
    $\begingroup$ I edited your post to emphasize the statistical question, showing it is not an R question. Please verify that the question still reflects what you intended to ask. $\endgroup$ – whuber Jun 9 '16 at 14:50
1
$\begingroup$

It sounds like you're doing the following:

  1. Given $n$ data points $\{X_1, ..., X_n\}$, calculate a pairwise cosine distance matrix $C$, where $C_{ij}$ is the cosine distance between points $i$ and $j$.
  2. Treat each row of $C$ as a vector. Find a Euclidean distance matrix $E$, where $E_{ij}$ is the euclidean distance between rows $i$ and $j$ of $C$
  3. Perform clustering, using $E$

This seems like an unusual thing to do, but I don't see any reason you can't do it.

One way to interpret your procedure is that you're doing clustering using 'higher order' distances (not sure if that's the correct terminology). The final distance between points $X_i$ and $X_j$ measures the discrepancy between the cosine distance from each of these points to all other points. According to this measure, $X_i$ and $X_j$ will be near when they have similar cosine distances to all other points.

Another way to interpret your procedure is that you're doing clustering in a feature space. You could think of step 1 above as mapping the original points into a feature space with $n$ dimensions (as many dimensions as data points). The projection of each point along dimension $j$ of the feature space is the cosine distance between that point and $X_j$. In steps 2 and 3, you do clustering using pairwise Euclidean distances in this feature space.

But, what was the error you got when trying to cluster using cosine distances?

$\endgroup$
  • $\begingroup$ Thank you. This really helped! When i tried passing the cosine matrix for the cluster, i got the error- Error in if (is.na(n) || n > 65536L) stop("size cannot be NA nor exceed 65536") : missing value where TRUE/FALSE needed. I guess, this may be because of very large data matrix from Cosine $\endgroup$ – Joe Jun 9 '16 at 18:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.