sentence tokenization with deep learning

Question

Anyone have any key words I can google, or links, for tutorials regarding how to do sentence tokenization using deep learning?

The machine learning MLE method looks like so much work you may as well just use deep learning. Basic sentence tokenization using heuristics isn't proficient for sentences like:

"He adds, in a far less amused tone, that the government has been talking about making Mt. Kuanyin a national park for a long time, and has banned construction or use of the mountain." -- (taken from https://tech.grammarly.com/blog/posts/How-to-Split-Sentences.html)

The only thing I've gotten remotely close via google is: http://campuspress.yale.edu/yw355/deep_learning/ (Deep Learning Project in NLP) which is a bunch of fluff with nothing concrete.

I'm looking for a basic how to project for someone like me how has dabbled only slightly with tensor-flow, but knows the basics of nueral nets and just started getting into LSTMs / RNNs. A book title... A web tutorial, something like that..

(I've found something released by google called SentencePiece , but it's kind of a black box at this point and I don't know what it does or if it would be useful)

Answer 1

My first question would be, why would you use deep learning for tokenization (performance and accuracy-wise)? Apart from some corner cases it's a pretty straightforward task. Existing statistics-based systems and rule-based methods are fast and quite accurate, unlike deep learning approaches which are computationally expensive and require large datasets to be trained. Squeezing extra, say 0.5% from tokenization for an arbitrary task in NLP may not be worth the effort. Check out Moses tokenizer or SpaCy's tokenizer, if you haven't given them a try yet.