I am working on Twitter dataset for emotion classification task. Often the tweets contain joined words. Some examples:

  1. #idontlikethis
  2. #worstdayever
  3. whatthehell

Sometimes the words are intentionally joined where the user explicitly joins them in a hashtag (example 1 and 2 above). Sometimes they might be unintentional, due to whatever reasons (example 3).

How should such words be handled in the preprocessing step?

One obvious way is to write a dictionary based solution to separate the words using statistical/probabilistic/HMM model. See this related answer on SO. But as the answer describes, and I agree, the problem itself is not that easy for a human without seeing the context.

I see two major challenges in tackling this problems:

1) Does the target word really contain joined words?

e.g The word waterworks has two possibilities.

Taught my students today how the waterworks. (joined words - water works)

Here comes the waterworks. What a baby. (not joined words - waterworks)

2) The target word has multiple possible splits.

e.g. the word wickedweather has 4 possibilities.

wicked weather
wicked we at her
wick ed weather
wick ed we at her

What is preferred way to solve this problem in text classification tasks? I can surely try a primitive way as suggested in the SO answer and see if my model's accuracy improves. But that's just trial and error strategy which I don't want to rely on.

How the researchers have been handling this problem? References to related literature is also highly appreciated.

  • $\begingroup$ You can look up the Morfessor python library that might help you do this. $\endgroup$ – Aaron Jun 18 '17 at 21:25

It's a bit too long to describe here, but Peter Norvig (of AI and Google fame) in the chapter called Natural Language Corpus Data from the book Beautiful Data by Segaran and Hammerbacher (2009) describes a statistical word segmentation algorithm for precisely this problem. The chapter is downloadable as PDF, and describes a working solution in Python on pages 1-9, so just take a look.

As this is CrossValidated, not StackOverflow, to also point to a more statistically oriented discussion of the solution, there is also a blog post by Sanket Patil describing the solution purely from its statistical view-point only (instead of a "computer coded" one).

In a nutshell, the approach is about finding the most likely segmentation by evaluating all possible segmentations as Markov chains with word probabilities taken from the Google n-gram corpus. And therefore this solution goes beyond the uni-gram approach described by the SO post/answer linked to in the question itself.

Notice that the solution should work for "tricky" cases as hinted in the question, too - but only in theory: the particular example with "waterworks" is a poor one where I bet even most humans would not always agree on when to split and when not, either.

For more common examples, you should get a probability that is higher for some "X AB" or "AB X" bi-gram than for the "A B" bigram, i.e., for "the waterworks" vs. "water works". So with Norvig's approach, as "the waterworks" has a much better probability than taking "the water" times "water works", the program always produces "the waterworks".

Therefore, to give you some proof that Norvig's solution indeed works for your cases, I suggest a few more practical examples (adapted from his book chapter):

>>> ngrams.segment2("insufficientnumbers")
(-8.715317997754482, ['in', 'sufficient', 'numbers'])
>>> ngrams.segment2("insufficientfunds")
(-6.976606271979648, ['insufficient', 'funds'])
>>> ngrams.segment2("choosespain")
(-8.35482244334036, ['choose', 'spain'])
>>> ngrams.segment2("choosespainkillers")
(-12.023363792731391, ['chooses', 'painkillers'])

Finally, in case its not obvious, Norvig's solution is only based on bi-grams; If you can get access to frequencies of higher n-grams (tri-grams, etc.), the approach would be able to take even more context into account and therefore could produce even better results.

Maye it should also be remarked that "whatthehell" is used so frequently that it is considered a valid single word by this approach. Depending on your use-case that might be desirable or not - but for a text classification task I would assume this behavior should work in your favor.

| cite | improve this answer | |

These words are generally termed as 'Superwords'. One approach that is generally taken, specially when you know such superwords would occur, is to consider character level features. For example,


will still be similar as these will have a lot of common character-n-gram features.

There is a more complicated approach though, inspired from this idea, to separate words in superwords.

Take all such superwords (hashtags in this case) and extract all char-tri-grams from them. You should now phrase detection in the hashtag-tri-gram corpus, considering char-tri-grams as words.

Section 4 in this paper will further explain on how to do phrase detection using words.

| cite | improve this answer | |
  • $\begingroup$ "Section 4" in Mikolov's (2013) word2vec paper describes collocation detection; This is actually something quite different, as I've described at length in another answer to another question. But still, just taking all character n-grams is a viable way of generating features for document classification without having to worry about word segmentation (though, in my personal opinion at least, maybe not the most effective approach). $\endgroup$ – fnl Jun 6 '17 at 22:20
  • $\begingroup$ "But still, just taking all character n-grams is a viable way of generating features for document classification". I agree with this. The only reason I discussed that character level approach was because the question specifically asked it. $\endgroup$ – silent_dev Jun 7 '17 at 10:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.