In the ML learning textbook I am working through, it says, that for NLP we construct a feature vector from the Text via the Bag of Words model. For that, we are using

from sklearn.feature_extraction.text import CountVectorizer
count = CountVectorizer(ngram_range=(1,1))
docs = np.array(["The sun is shining", 
                 "The weather is sweet", 
                 "The sun is shining and the weather is sweet"])
bag = count.fit_transform(docs)

From this we can get an array with the number of words for each Sentence and a dictionary to look up the indices of each word in this array

Now later, the book says, we also need a Tokenizer like this

def tokenizer(text):
     return text.split()

(or optional with a stemmer) which simply takes a text, and splits it into an array, where each element contains a word. I don't really get why we need to do this though, since if we apply the tokenizer first, we just have a long array of words, and not as in the example of the CountVectorizer, an array of sentences


1 Answer 1


A CountVectorizer first tokenizes the text, then count tokens. It therefore needs to have a Tokenizer, as you can see for example in sklearn.feature_extraction.text.CountVectorizer, which takes a tokenizer as argument :

class sklearn.feature_extraction.text.CountVectorizer(input=u'content',
encoding=u'utf-8', decode_error=u'strict', strip_accents=None, 
lowercase=True, preprocessor=None, tokenizer=None, stop_words=None, 
token_pattern=u'(?u)\b\w\w+\b', ngram_range=(1, 1), analyzer=u'word', 
max_df=1.0, min_df=1, max_features=None, vocabulary=None, binary=False, 
dtype=<type 'numpy.int64'>)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.