I'm working on a text classification problem using Python and NLTK. I've got two frequency distributions, one for each class (it's basically a binary classification). So, my doubt it's if there's a way to apply feature selection since i've got two separated models and the classificator algorithm was manually implemented.
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Words are problematic features.
- There are plenty of words so feature selection is needed.
- The common word are probably not what you are looking for (e.g., stop words) and those of interest are quite rare.
- Sometimes different words have similar meaning (e.g., buy-bought, happy-pleased).
- Sometime the word meaning might change due to context (e.g., happy-not happy-will be happy, white- the white house).
Working with the words as is (e.g., bag of words) or doing feature selection on them might be problematic.
Usually the goal of the analysis and properties of the text determine which methods you'll find useful.
You can combine words of similar meaning using stemming and by adding semantic data (e.g. word net).
You can use the context in order to differ between occurrences of the same word with different meanings. (e.g., n-grams).
Once you aggregated similar words into one feature and differ by context occurrences of the words both your classifier and feature selector should work better.