How to Handle Text Data in Regression?

Question

I'm trying to predict an article's page views based on its content and a bunch of other factors. Everything except for the content is numeric (e.g. time of day, length of title, length of article).

I was reading about one-hot encoding and it seems I can treat the textual content of the article as a "bag of words." Then I can extract features like:

hasSpecificWord: 1

hasOtherWord: 0

Alternatively, I could use the counts of those words as a numeric feature.

Is there a standard way of approaching this problem in ML?

Thanks!

Answer 1

I'd try CountVectorizer() from sklearn that does this job of converting into bag of words.

Ex:

from sklearn.feature_extraction.text import CountVectorizer 
import pandas as pd

 # Sample dataset
simple_train = ['Lets play', 'Game time today', 'This game is just awesome!']

#initialize the Vectorizer
vect = CountVectorizer()

# learn the vocab and parse them as features based on the given params.
vect.fit(simple_train)

# get the feature names
vect.get_feature_names()
['awesome', 'game', 'is', 'just', 'lets', 'play', 'this', 'time', 'today']

# convert the array into a df
df = pd.DataFrame(dtm.toarray(),columns=vect.get_feature_names()) 
df

 awesome    game    is  just    lets    play    this    time    today
0   0   0   0   0   1   1   0   0   0
1   0   1   0   0   0   0   0   1   1
2   1   1   1   1   0   0   1   0   0

I think you've already taken care of this: Including other aspects of the article such as source rating, search engine ranking of the source site, categories, tags as features for the model along with the aspects you mentioned such as (e.g. time of day, length of title, length of article).

Answer 2

As far as I know, pretty standard approach is using term vectors - just like you said. Algo is roughly

1. Clean text from stop words (i.e. articles)
2. Normalize your data with stemmer
3. Map your observation text via dictionary (which must be stemmed beforehand with the same stemmer)

Sometimes you don't even need to form vector space by word count, but rather with presence of words (effectively, you may end up with having {0, 1} vectors). The latter works out surprisingly good often.

The dictionary to map your article may be top N frequent words from basic dictionary or may be collected from previous articles/similar articles on the web, if you can build or collect such corpus