1
$\begingroup$

As described here and there, Chi2 feature selection can be used in order to filter a set of words that have a low dependency with a given class C.

In my case, I have a corpus of documents where each is assigned a sentiment (positive or negative). After pre-processing my text (stopwords, stemming and removing words that appear in less than 1%/more than 50% of my documents), I define my positive class as any word contained in a document with positive sentiment, and same for the negative one.

I then use the Chi2 filtering approach with contingency tables. Below is an example of for word 'increase' and my positive class:

enter image description here

The problem I have is that my 2 classes are quite broad (hence the Chi2 filtering!) and there's always a word from each class in any document. Which makes my N01 and N00 equal to 0 for all the words.

The denominator of my Chi2 formula (below) is then systematically equal to 0 because of the 4th term.

enter image description here

Is there something that I'm not doing well? If not, is there a trick or an alternative I could use to avoid this problem?

Many thanks!

$\endgroup$
0
$\begingroup$

Chi-square assumes that you have a large enough expected value of each cell count $k \geq 5$. This means that each cell of the contingency table should have a minimal value of $5$.

$\endgroup$
  • 2
    $\begingroup$ Welcome to Stats.SE. I've edited your answer to fix the typo (there is an edit button below your answer) - it should appear in a few minutes (after that, you may delete your comment since it's no longer needed). By the way, take the opportunity to take the tour( stats.stackexchange.com/tour), if you haven't done it already. See also some tips on formatting help and write down equations using LaTeX / MathJax. $\endgroup$ – Ertxiem Jul 10 at 18:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.