1
$\begingroup$

I wish to do topic modeling on text corpus some of which are about company earnings which has lots of numbers in it. It has no sentence structure. I think tagging numbers using nltk.pos_tagging can help me find out if the number is CD (numeral/cardinal). Using numeral feature as one of the many features in the BOW can help me identify the topic related to Statistics or Maths. Intuitively if I drop this information about numbers appearing in the text, I will not be able to figure out the topic of unseen texts.

I will really appreciate your views regarding this. Also, if anyone knows about a BOW library in python that can do so internally that will be helpful. I came across scikit-learn library's count vectorizer in which I think if I change token_pattern, it might help me in getting numbers as features but that will not help me in collapsing all numeric features as one feature.

I hope I am clear. Really appreciate your time and help.

Thanks.

$\endgroup$
  • 1
    $\begingroup$ I would have try to do it in a pre-processing step. Use a regular expression in order to convert all numbers into a single symbol. After that, continue regularly. Be sure to use a BOW library that count the terms and not just nots their existence. Even so, If the ratio of numbers in the text was the only feature, I guess it is correlated to math but won't enable too good prediction. $\endgroup$ – DaL Aug 25 '16 at 10:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.