I have a set of English and foreign language documents that I would to perform k-means clustering on to find document groups by topic. These documents are concatenated social media comments for individual users, which may be 1000-5000 words. (For example, a document may contain every question user 'Paul' has asked on StackExchange.)

I have performed feature extraction in two ways:

(1) extracting entity pronouns from a known list, which proved to have too poor coverage on the document set

(2) raw tokenisation, which proved too noisy with my poor quality feature selection

My usual strategy for feature selection for supervised learning is to calculate TF/IDF scores and systematically try different thresholds to provide the best cross validation performance. (Clearly impossible here.)

I have TF/IDF scores and I've tried using intuition to select a good threshold, but I'm struggling to evaluate if the clustering is good or bad. Google has returned some papers but nothing that is as prescriptive as I would like.

Any suggestions on getting started with feature selection for k-means or other unsupervised clustering?

Many thanks.


2 Answers 2


Dangers of TF-DIF on different languages

First of all, as stressed by @Roosh a TF-IDF on documents having documents in different languages seems to be a bad idea to me.

Imagine the case where two languages do not share the same alphabet (like Japanese and English). In the TF-IDF embedding, they will leave on different axises. And the same thing would happen for language who do not share (m)any words (like French and German).

So my first impression is that unsupervised clustering on documents in different languages will create clusters mostly related to the languages themselves.

Another approach

Therefore you may want to use a different embedding than TFIDF. Word2Vec (described in Distributed Representations of Words and Phrases and their Compositionality) has been applied with two different languages simultaneously: Joint word2vec Networks for Bilingual Semantic Representations


You will in any case need a single basis for working with texts. Try first automated translated foreign language document to English. If only foreign language problem, it can give a good approximation.

How much longer texts do you use? If shorten, what is T in your TF/IDF - word? How about try use ngramm - it can provide higher accuracy. What foreign languages? If this is the Latin group language, the use stemming and subsequent use ngramm can give a good result.

It's hard to say without seeing the data about what signs will help achieve better results.

  • $\begingroup$ Thanks, you raise some good points. I updated my question to give more information. I'm beginning to think my feature extraction is poor in the first place. $\endgroup$
    – Paul
    Jan 23, 2015 at 8:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.