# Why do we need Tokenzier if we have Vectorizer

In the ML learning textbook I am working through, it says, that for NLP we construct a feature vector from the Text via the Bag of Words model. For that, we are using

from sklearn.feature_extraction.text import CountVectorizer
count = CountVectorizer(ngram_range=(1,1))
docs = np.array(["The sun is shining",
"The weather is sweet",
"The sun is shining and the weather is sweet"])
bag = count.fit_transform(docs)


From this we can get an array with the number of words for each Sentence and a dictionary to look up the indices of each word in this array

Now later, the book says, we also need a Tokenizer like this

def tokenizer(text):
return text.split()


(or optional with a stemmer) which simply takes a text, and splits it into an array, where each element contains a word. I don't really get why we need to do this though, since if we apply the tokenizer first, we just have a long array of words, and not as in the example of the CountVectorizer, an array of sentences

A CountVectorizer first tokenizes the text, then count tokens. It therefore needs to have a Tokenizer, as you can see for example in sklearn.feature_extraction.text.CountVectorizer, which takes a tokenizer as argument :
class sklearn.feature_extraction.text.CountVectorizer(input=u'content',