# Tf-idf for text classification: On what should IDF be calculated?

The TF-IDF value of a word specifies how important a word for each document is. My setting is any text classification where one has multiple documents of with different classes:
Let's take a lot of movie reviews with a feature 'sentiment' which is 0 or 1 (negative or positive).

The goal of the classification would be to classify a new review as either 0 or 1.
Approach 1: My idea behind using TF-IDF here, is to understand if a word is important for the set of reviews with the sentiment 0 or the set of reviews with the sentiment 1. This means that I would calculate the IDF values based for two documents (because of two classes) where I put together all the reviews of the same class, meaning:

1. Doc1: The text of all reviews with sentiment 0
2. Doc2: The text of all reviews with sentiment 1

I have looked into a few examples and this approach is not what I see done in them.
Approach 2: In the examples for using TF-IDF for text classification, the IDF-values are calculated for each review and not each class (see here), meaning the number of documents is the number of reviews.

Q: Which approach is correct?

My 'pipeline' of classification via tfidf would be:

1. Calculate the IDF with the documents being every review concatenated for each class in all training reviews (like in Doc1 and Doc2)
2. Calculate the TF for every word in each training review
3. Calculate the TF for every word in each testing review
4. Multiply TF*IDF to get the TFIDF-matrix for either train or test

Here is an example in R to illustrate both approaches:
(train_tfidf follows my idea and example_train_tfidf follows approach 2. I calculate the tfidfs, train a model for each approach, make predictions and compare AUCs)

library(text2vec)
library(glmnet)

data("movie_review")
train <- movie_review[1:4000, ]
test <- movie_review[4001:5000, ]

# My idea behind TF-IDF
# create a tfidf-model with one document for each class
create_tfidf_model <- function(train){
# This is the crucial step for my approach!
train <- aggregate(review ~ sentiment, data = train, FUN = paste, collapse = " ")
train_tokens <- itoken(train$review, preprocessor = tolower, tokenizer = word_tokenizer, ids = train$id, progressbar = FALSE)
train_vocab <- create_vocabulary(train_tokens, stopwords = stopwords::stopwords("en"))
vectorizer <- vocab_vectorizer(train_vocab)
train_dtm <- create_dtm(train_tokens, vectorizer)
tfidf_model <- TfIdf$new() # fit model train_tfidf <- fit_transform(train_dtm, tfidf_model) return(list(tfidf_model, vectorizer, train_tfidf)) } # Take a vectorizer and (tf)idf-model and calculate a tfidf-Matrix for data with the idf-weights of the tfidf-model for the words in the vectorizer create_tfidf_from_model <- function(data, tfidf_model, vectorizer){ data_tokens <- itoken(data$review,
preprocessor = tolower, tokenizer = word_tokenizer, ids = data$id, progressbar = FALSE) data_dtm <- create_dtm(data_tokens, vectorizer) data_tfidf <- transform(data_dtm, tfidf_model) return(data_tfidf) } # Example/Approach 2 from https://cran.r-project.org/web/packages/text2vec/vignettes/text-vectorization.html tfidf_example <- function(train){ train_tokens <- itoken(train$review,
preprocessor = tolower, tokenizer = word_tokenizer, ids = train$id, progressbar = FALSE) train_vocab <- create_vocabulary(train_tokens, stopwords = stopwords::stopwords("en")) vectorizer <- vocab_vectorizer(train_vocab) train_dtm <- create_dtm(train_tokens, vectorizer) tfidf_model <- TfIdf$new()
train_tfidf <- fit_transform(train_dtm, tfidf_model)
return(list(tfidf_model, vectorizer, train_tfidf))
}

## Prediction
tfidf_model <- create_tfidf_model(train)
train_tfidf <- create_tfidf_from_model(train, tfidf_model = tfidf_model[], vectorizer = tfidf_model[])
test_tfidf <- create_tfidf_from_model(test, tfidf_model = tfidf_model[], vectorizer = tfidf_model[])

example_tfidf_model <- tfidf_example(train)
example_train_tfidf <- create_tfidf_from_model(train, tfidf_model = example_tfidf_model[], vectorizer = example_tfidf_model[])
example_test_tfidf <- create_tfidf_from_model(test, tfidf_model = example_tfidf_model[], vectorizer = example_tfidf_model[])

# Train models
glmnet_clas_mine <- cv.glmnet(x = train_tfidf, y = train[['sentiment']],
family = 'binomial', alpha = 1, type.measure = "auc", nfolds = 4, thresh = 1e-3, maxit = 1e3)

glmnet_clas_example <- cv.glmnet(x = example_train_tfidf, y = train[['sentiment']],
family = 'binomial', alpha = 1, type.measure = "auc", nfolds = 4, thresh = 1e-3, maxit = 1e3)

plot(glmnet_clas_mine)
print(paste("max AUC =", round(max(glmnet_clas_mine$cvm), 4))) plot(glmnet_clas_example) print(paste("max AUC =", round(max(glmnet_clas_example$cvm), 4)))

# predict with models
pred <- predict(glmnet_clas_mine, test_tfidf, type = 'response')[,1]
example_pred <- predict(glmnet_clas_example, example_test_tfidf, type = 'response')[,1]
# AUC
glmnet:::auc(test$sentiment, pred) glmnet:::auc(test$sentiment, example_pred)


Evaluating the prediction on the test set gives me an AUC of 0.905 with my approach and an AUC of 0.912 where I count each review as one document. The AUCs are very close to each other. But I my approach is slightly worse.
(The same results holts on the AUC on the train set)

But there is a one thing which I noticed:

# Their sparsity is about equal, but my approach has more values which equal 0
sum(train_tfidf@x != 0)
sum(example_train_tfidf@x != 0)

# This should normally not be done as the matrices are big and sparse
train_tfidf <- as.data.frame(as.matrix(train_tfidf))
example_train_tfidf <- as.data.frame(as.matrix(example_train_tfidf))

# Remove columns where every entry is 0
train_tfidf <-  train_tfidf[, colSums(train_tfidf != 0) > 0]
example_train_tfidf <-  example_train_tfidf[, colSums(example_train_tfidf != 0) > 0]
ncol(train_tfidf)
ncol(example_train_tfidf)


My approach has approximately half the number of features (~20k less) after removing features where every entry is 0 which sounds good to me. But I am still confused why Approach 2 is used in every example. Maybe because it is less code to do and easier to give an example of it?