7
$\begingroup$

Term frequency - inverse document frequency is term count within a document weighted against the term's ubiquity within the corpus. This weight is based on the principle that terms occurring in almost every document are therefore less specific to an individual document and should be scaled down so that a tf-idf value represents the term's relative importance within a document.

However, won't an SVM using a linear classifier appropriately weight term frequency within a document? Wouldn't it automatically account for ubiquitous terms and decrease their weight respectively?

$\endgroup$

1 Answer 1

4
$\begingroup$

If you only multiply each feature by some weight that correspond to the term's rarity (i.e. $log(\frac{M}{a_i})$ where $M$ is the total number of documents and $a_i$ is the number of documents with the considered term), and then use SVM, then the feature scaling you have performed is useless (as you have observed).

However, if after the scaling you also normalize your data, and then perform an SVM,then you get different result from what you would get, if you simply used SVM or used SVM on normalized data without feature scaling. This can have possibly positive effects, for two reasons:

1) Normalization sounds reasonable, because word counts would be very different for long and short document, while normalized word counts reflect the frequency of the word in the document, and it's importance.

2) If you perform the feature scaling by rarity terms before normalization, the normalized vectors will be longer in the direction of rare words, which are possibly of bigger importance for distinguishing between documents.

$\endgroup$
2
  • $\begingroup$ All the approaches I've seen for tf-idf normalize the term count into term frequency before the idf weighting. I'm not sure why this approach is taken instead of yours, but it makes me wonder. Thanks though. This is a great answer. $\endgroup$ Aug 22, 2012 at 4:26
  • $\begingroup$ what do you mean by normalization? won't that again be scaling of each feature by a linear factor? $\endgroup$
    – ihadanny
    Sep 21, 2016 at 7:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.