# When parsing text for n-grams - should punctuation be included?

I want to start working on data-mining by parsing text. It seems like the best place to start is by processing n-grams out of text to try sentiment analysis.

Muffins are fine, I wouldn't say I like them though.


However, I'm curious to know if I should include punctuation or not. (I plan on starting with 3-grams and working up since I'm not sure 2-grams include enough information for accurate results.)

Muffins are fine | are fine [,] | I wouldn't say | ....


Since a "," was found, start over at next word after the ",". Instead of including the punctuation like normal.

Muffins are fine | are fine , | fine , I | , I wouldn't | ...


Can anyone tell me if this is a bad idea?

Google ignores punctuation, but there are non-alphanumeric characters that are not ignored.

For example, search these word/phrases:

• tech spec
• tech-spec
• tech,spec
• tech_spec

The search results vary, showing Google does consider some characters significant.

Also, are you doing this in non-English languages?

If so, then consider creating n-grams from a certain number of characters, instead of words. This will lead to better results on many non-English languages, and it's the only way to effectively parse CJK-type languages that don't use significant whitespace.

Here's a hint: Google doesn't include punctuation in n-grams.

• +1 That's some hint. Do you know if they just skip to the next word, or do they just stop there? – Xeoncross Feb 17 '12 at 0:11
• They just skip to the next word. – Carlos Accioly Feb 17 '12 at 0:18
• Actually, I asked that incorrectly. Do they add the next word to the current set or just end the current set and start over fresh on the next word? – Xeoncross Feb 17 '12 at 0:23
• It's unlikely that Google just skips to the next 'word' when they arrive at characters that are actually discarded. Here's a test, search for "tech spec" and "t-e-c-h-s-p-e-c" and see how the results compare. After all, the idea of discarding characters is based on statistics showing that people may not use them accurately or consistently enough to be considered significant. – Mike Bijon Mar 19 '12 at 17:24