1
$\begingroup$

Please bear with me if the question sounds stupid.

Application/Desire : i want to be able to cluster word2vec vectors using density based clustering algorithms (say dbscan/hdbscan; due to too much noise in data) using python or R. I cannot compute pairwise distance b/w vectors as the size is too big (>2.5 million vocab). DBSCAN/HDBSCAN in both R and python does not directly support cosine distance as a metric.

Question: Using the vectors (say 250 dimensions), if i am to reduce it using a non-linear dim reduction algo like T-SNE or autoencoder or SOM to say 50 dimensions, can i use euclidean metric to cluster using density based clustering algos?? does the dim reduction algos also shift distance metrics? so that i can here use euclidean instead of cosine metric ??

Other suggestions are also welcome

$\endgroup$
1
$\begingroup$

For all I know, sklearn DBSCAN does support cosine.

There also is ELKI which I have used with Cosine and DBSCAN. You can add an index (e.g. Cover tree for arc cosine) to accelerate DBSCAN. It's very fast, and scalable. It often still works when the others (in particular sklearn) run out of memory if you set the Java memory limit parameter -Xmx.

But at 2.5 million points runtime may be several hours nevertheless. And you will need many iterations to tune parameters. You should consider sampling. At the least, use a sample to tune parameters and validate your approach first.

As for tSNE and clustering, K-means clustering on the output of t-SNE is a very visual explanation why tSNE should only be used for visualization and not for clustering. (Short story: it preserves some neighbors, but neither distances nor densities; so anything that is distance or density based like clustering must not be done afterwards).

$\endgroup$
  • $\begingroup$ Thanks for your reply Anony.. I am newbie when it comes to Java programming. I understand that this is beyond the question that i asked just now. But do you have a working code for HDBSCAN in JAVA? or can you guide me where i can find some working code for HDBSCAN with cosine distance ? $\endgroup$ – Sundaresh Prasanna Oct 16 '17 at 9:55
  • $\begingroup$ ELKI already mentioned above also appears to have HDBSCAN* with support for cosine. $\endgroup$ – Anony-Mousse Oct 16 '17 at 15:06
  • $\begingroup$ Can autoencoders be used on word2vec vectors prior to clustering? $\endgroup$ – Enthusiast Jan 31 '18 at 6:10
  • $\begingroup$ To solve what mathematical problem? Don't stack functions just because they sound cool. $\endgroup$ – Anony-Mousse Jan 31 '18 at 7:13
  • $\begingroup$ word2vec gives a dense vector. I want to reduce the dimensionality of the vector using autoencoder. Then use the reduced vectors for DBSCAN clustering. $\endgroup$ – Enthusiast Jan 31 '18 at 9:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.