I am working on Twitter dataset for emotion classification task. Often the tweets contain joined words. Some examples:
Sometimes the words are intentionally joined where the user explicitly joins them in a hashtag (example 1 and 2 above). Sometimes they might be unintentional, due to whatever reasons (example 3).
How should such words be handled in the preprocessing step?
One obvious way is to write a dictionary based solution to separate the words using statistical/probabilistic/HMM model. See this related answer on SO. But as the answer describes, and I agree, the problem itself is not that easy for a human without seeing the context.
I see two major challenges in tackling this problems:
1) Does the target word really contain joined words?
e.g The word
waterworks has two possibilities.
Taught my students today how the waterworks. (joined words - water works)
Here comes the waterworks. What a baby. (not joined words - waterworks)
2) The target word has multiple possible splits.
e.g. the word
wickedweather has 4 possibilities.
wicked we at her
wick ed weather
wick ed we at her
What is preferred way to solve this problem in text classification tasks? I can surely try a primitive way as suggested in the SO answer and see if my model's accuracy improves. But that's just trial and error strategy which I don't want to rely on.
How the researchers have been handling this problem? References to related literature is also highly appreciated.