2
$\begingroup$

I am working on textual dataset containing data from official documents like reports by companies, legal documents, speeches by directors to shareholders etc. The content of textual documents are like:

  1. In respect of its fixed assets: a. The Company has maintained proper records showing full particulars including quantitative details and situation of fixed assets.

Your Company owns 99.99 percent of 20 Microns Sdn. Bhd. During the year under review, the said Company reported Gross turnover Rs. 295.74 Lacs

There a lot's of period which donot represents the sentence boundary. Until now i have implemented the regular expression based sentence segmentation by designing some rules and considerably good performance. But still it need to be updated when new words with period occur in the sentences and which seems like never ending task. I have searched and somewhere it is recommended to have Maximum Entropy classifier based word segmentation. I don't get exactly how to use MaxEnt classifier for word segmentation. How do i implement machine learning based word segmentation i.e based on maximum entropy classifier or any other which performs better than regular expression thing?

$\endgroup$
2
$\begingroup$

For an ML-based solution you generally would want a sequence learner like a conditional random field or a recursive neural network. Now each symbol has a binary label for classification representing whether the current symbol is a sentence boundary or not. Supervised learning on such data will result in your required classifier. You could theoretically also try a sliding window approach on feed forward architectures, although I'd expect it to perform quite a bit worse.

Perhaps have a look at the relevant parts of NLTK, if you want an overview of various sentence tokenization methods: http://www.nltk.org/api/nltk.tokenize.html

Here is an example of how to perform supervised sentence segmentation: http://www.nltk.org/book/ch06.html

$\endgroup$
  • 1
    $\begingroup$ Addendum: The punkt tokenizer works in an unsupervised fashion, if you are interested in that: nltk.org/_modules/nltk/tokenize/punkt.html $\endgroup$ – Martin Krämer Jan 25 '17 at 12:30
  • $\begingroup$ But it is written in documentation of punkt sentence tokenizer "It must be trained on a large collection of plaintext in the target language before it can be used." What kind(structure) of training data-set it required for training? $\endgroup$ – Sanjeev Jan 25 '17 at 18:30
  • $\begingroup$ It basically implements this paper: linguistics.ruhr-uni-bochum.de/~kiss/publications/… So it simply requires lots of text in the language you want to train it on. But if you just want to apply it I am pretty sure that NLTK comes with a pretrained model. $\endgroup$ – Martin Krämer Jan 25 '17 at 18:32
  • $\begingroup$ It is working good with pre-trained model. But in the data-set there are a lot of Rs. 101.2 / Section No. / No. It detects these '.' as the sentence boundary. Is there any way to add these in per-trained model. The way i am considering to counter this problem is that remove '.' after such words using regular expression and then apply sentence segmentation or is there any way to perform for '.' which are not in trained models. $\endgroup$ – Sanjeev Jan 25 '17 at 18:53
  • 1
    $\begingroup$ Find out how to tweak punkt sentence tokenizer at stackoverflow.com/questions/14095971/… . Thanks . $\endgroup$ – Sanjeev Jan 25 '17 at 19:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.