How to handle joined words in text classification tasks?

Question

I am working on Twitter dataset for emotion classification task. Often the tweets contain joined words. Some examples:

1. #idontlikethis
2. #worstdayever
3. whatthehell

Sometimes the words are intentionally joined where the user explicitly joins them in a hashtag (example 1 and 2 above). Sometimes they might be unintentional, due to whatever reasons (example 3).

How should such words be handled in the preprocessing step?

One obvious way is to write a dictionary based solution to separate the words using statistical/probabilistic/HMM model. See this related answer on SO. But as the answer describes, and I agree, the problem itself is not that easy for a human without seeing the context.

I see two major challenges in tackling this problems:

1) Does the target word really contain joined words?

e.g The word waterworks has two possibilities.

Taught my students today how the waterworks. (joined words - water works)

Here comes the waterworks. What a baby. (not joined words - waterworks)

2) The target word has multiple possible splits.

e.g. the word wickedweather has 4 possibilities.

wicked weather
wicked we at her
wick ed weather
wick ed we at her

What is preferred way to solve this problem in text classification tasks? I can surely try a primitive way as suggested in the SO answer and see if my model's accuracy improves. But that's just trial and error strategy which I don't want to rely on.

How the researchers have been handling this problem? References to related literature is also highly appreciated.

Answer 1

It's a bit too long to describe here, but Peter Norvig (of AI and Google fame) in the chapter called Natural Language Corpus Data from the book Beautiful Data by Segaran and Hammerbacher (2009) describes a statistical word segmentation algorithm for precisely this problem. The chapter is downloadable as PDF, and describes a working solution in Python on pages 1-9, so just take a look.

As this is CrossValidated, not StackOverflow, to also point to a more statistically oriented discussion of the solution, there is also a blog post by Sanket Patil describing the solution purely from its statistical view-point only (instead of a "computer coded" one).

In a nutshell, the approach is about finding the most likely segmentation by evaluating all possible segmentations as Markov chains with word probabilities taken from the Google n-gram corpus. And therefore this solution goes beyond the uni-gram approach described by the SO post/answer linked to in the question itself.

Notice that the solution should work for "tricky" cases as hinted in the question, too - but only in theory: the particular example with "waterworks" is a poor one where I bet even most humans would not always agree on when to split and when not, either.

For more common examples, you should get a probability that is higher for some "X AB" or "AB X" bi-gram than for the "A B" bigram, i.e., for "the waterworks" vs. "water works". So with Norvig's approach, as "the waterworks" has a much better probability than taking "the water" times "water works", the program always produces "the waterworks".

Therefore, to give you some proof that Norvig's solution indeed works for your cases, I suggest a few more practical examples (adapted from his book chapter):

>>> ngrams.segment2("insufficientnumbers")
(-8.715317997754482, ['in', 'sufficient', 'numbers'])
>>> ngrams.segment2("insufficientfunds")
(-6.976606271979648, ['insufficient', 'funds'])
>>> ngrams.segment2("choosespain")
(-8.35482244334036, ['choose', 'spain'])
>>> ngrams.segment2("choosespainkillers")
(-12.023363792731391, ['chooses', 'painkillers'])

Finally, in case its not obvious, Norvig's solution is only based on bi-grams; If you can get access to frequencies of higher n-grams (tri-grams, etc.), the approach would be able to take even more context into account and therefore could produce even better results.

Maye it should also be remarked that "whatthehell" is used so frequently that it is considered a valid single word by this approach. Depending on your use-case that might be desirable or not - but for a text classification task I would assume this behavior should work in your favor.

Answer 2

These words are generally termed as 'Superwords'. One approach that is generally taken, specially when you know such superwords would occur, is to consider character level features. For example,

#idontlikethis
#idonotlikethis

will still be similar as these will have a lot of common character-n-gram features.

There is a more complicated approach though, inspired from this idea, to separate words in superwords.

Take all such superwords (hashtags in this case) and extract all char-tri-grams from them. You should now phrase detection in the hashtag-tri-gram corpus, considering char-tri-grams as words.

Section 4 in this paper will further explain on how to do phrase detection using words.