How do I weight words in title, body text, and links differently in document clustering?

Question

I'm currently trying to play around with NLTK and scikits-learn for text clustering news articles.

How do I extend the models to add the scaling of features from a document (I'm doing some preprocessing on the text articles) so I can experiment by weighting ?

I'm starting from this outline of document clustering:

https://github.com/ogrisel/scikit-learn/blob/master/examples/document_clustering.py

How do I approach this problem? Do I add develop heuristics to help tune the parameters I give kmeans?

a. Title b. Body Text c. Links (anchor text and link)

Thanks.

Answer 1

One simple (but often effective) solution is to simply duplicate the different sections according to the weights. For example, if you include two copies of the title and links and one copy of the body, then the term frequencies of the terms in the titles and links will be doubled.