2
$\begingroup$

Application/Desire : I want to be able to cluster word2vec vectors using density based clustering algorithms (say dbscan/hdbscan; due to too much noise in data) using python or R. I cannot compute pairwise distance b/w vectors as the size is too big (>2.5 million vocab). DBSCAN/HDBSCAN in both R and python does not directly support cosine distance as a metric.

Question: Using the vectors (say 250 dimensions), if I am to reduce it using a non-linear dim reduction algo like T-SNE or autoencoder or SOM to say 50 dimensions, can I use euclidean metric to cluster using density based clustering algos? does the dim reduction algos also shift distance metrics? so that I can here use euclidean instead of cosine metric ??

Other suggestions are also welcome

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

For all I know, sklearn DBSCAN does support cosine.

There also is ELKI which I have used with Cosine and DBSCAN. You can add an index (e.g. Cover tree for arc cosine) to accelerate DBSCAN. It's very fast, and scalable. It often still works when the others (in particular sklearn) run out of memory if you set the Java memory limit parameter -Xmx.

But at 2.5 million points runtime may be several hours nevertheless. And you will need many iterations to tune parameters. You should consider sampling. At the least, use a sample to tune parameters and validate your approach first.

As for tSNE and clustering, K-means clustering on the output of t-SNE is a very visual explanation why tSNE should only be used for visualization and not for clustering. (Short story: it preserves some neighbors, but neither distances nor densities; so anything that is distance or density based like clustering must not be done afterwards).

Improve this answer
$\endgroup$
8
  • $\begingroup$ Thanks for your reply Anony.. I am newbie when it comes to Java programming. I understand that this is beyond the question that i asked just now. But do you have a working code for HDBSCAN in JAVA? or can you guide me where i can find some working code for HDBSCAN with cosine distance ? $\endgroup$
    – Sundaresh Prasanna
    Oct 16, 2017 at 9:55
  • $\begingroup$ ELKI already mentioned above also appears to have HDBSCAN* with support for cosine. $\endgroup$
    – Has QUIT--Anony-Mousse
    Oct 16, 2017 at 15:06
  • $\begingroup$ Can autoencoders be used on word2vec vectors prior to clustering? $\endgroup$
    – StatguyUser
    Jan 31, 2018 at 6:10
  • $\begingroup$ To solve what mathematical problem? Don't stack functions just because they sound cool. $\endgroup$
    – Has QUIT--Anony-Mousse
    Jan 31, 2018 at 7:13
  • $\begingroup$ word2vec gives a dense vector. I want to reduce the dimensionality of the vector using autoencoder. Then use the reduced vectors for DBSCAN clustering. $\endgroup$
    – StatguyUser
    Jan 31, 2018 at 9:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.