Why do we need Tokenzier if we have Vectorizer

Question

In the ML learning textbook I am working through, it says, that for NLP we construct a feature vector from the Text via the Bag of Words model. For that, we are using

from sklearn.feature_extraction.text import CountVectorizer
count = CountVectorizer(ngram_range=(1,1))
docs = np.array(["The sun is shining", 
                 "The weather is sweet", 
                 "The sun is shining and the weather is sweet"])
bag = count.fit_transform(docs)

From this we can get an array with the number of words for each Sentence and a dictionary to look up the indices of each word in this array

Now later, the book says, we also need a Tokenizer like this

def tokenizer(text):
     return text.split()

(or optional with a stemmer) which simply takes a text, and splits it into an array, where each element contains a word. I don't really get why we need to do this though, since if we apply the tokenizer first, we just have a long array of words, and not as in the example of the CountVectorizer, an array of sentences

Answer 1

A CountVectorizer first tokenizes the text, then count tokens. It therefore needs to have a Tokenizer, as you can see for example in sklearn.feature_extraction.text.CountVectorizer, which takes a tokenizer as argument :

class sklearn.feature_extraction.text.CountVectorizer(input=u'content',
encoding=u'utf-8', decode_error=u'strict', strip_accents=None, 
lowercase=True, preprocessor=None, tokenizer=None, stop_words=None, 
token_pattern=u'(?u)\b\w\w+\b', ngram_range=(1, 1), analyzer=u'word', 
max_df=1.0, min_df=1, max_features=None, vocabulary=None, binary=False, 
dtype=<type 'numpy.int64'>)