TF-IDF Calculation in Scikit Learn

Question

#Code from the NLP with Pytorch O'Reilly book
from sklearn.feature_extraction.text import TfidfVectorizer
import seaborn as sns
tfidf_vectorizer = TfidfVectorizer()
corpus = ['Time flies flies like an arrow.','Fruit flies like a banana.']
vocab = ['an','arrow','banana','flies','fruit','like','time']
tfidf = tfidf_vectorizer.fit_transform(corpus).toarray()

My resulting 'tfidf' array is (rounded for visual clarity):

array(
[[0.43, 0.43, 0.00, 0.61, 0.00 ,0.3, 0.43],
[0.00, 0.00, 0.58, 0.41, 0.58, 0.41, 0.00]])

The order of the words is:
['an','arrow','banana','flies','fruit','like','time']

I'm confused as to why my array gives different scores for the word 'flies' in both sentences. And why it gives it a score at all? Shouldn't the IDF = 0? Because Ndocuments = 2, Ndocument_with_word = 2, log(2/2) = 0

I understand the scores get normalized to be between 0 and 1, but why wouldn't they normalize to the same score, and why aren't they zero?

I did read through several other posts but wasn't able to find my answer, please don't close this as a copy as I'm unable to find an answer and am learning this from a book

Answer 1

So to start with there are some default options inside TfidfVectorizer() that you are using without realising. To make things line up with what you expect you should use

tfidf_vectorizer = TfidfVectorizer(norm=None, smooth_idf=False)

Using this option the score computed will be $$s_{ij} = tf_{ij}(1+log(N/df_{i})$$ where $s_{ij}$ is the score for the word i in document j, $tf_{ij}$ is the number of times word i appears in document j, N is the total number of documents and $df_{i}$ is the number of documents containing at least one instance of word i.

The default value of norm='l2' takes your scores and normalises them per document so that $\sum_i s_{ij}=1$

The default value of smooth_idf=True replaces the log above with $log((N+1)/(df_{i}+1))$