Tf-idf for text classification: On what should IDF be calculated?

Question

The TF-IDF value of a word specifies how important a word for each document is. My setting is any text classification where one has multiple documents of with different classes:
Let's take a lot of movie reviews with a feature 'sentiment' which is 0 or 1 (negative or positive).

The goal of the classification would be to classify a new review as either 0 or 1.
Approach 1: My idea behind using TF-IDF here, is to understand if a word is important for the set of reviews with the sentiment 0 or the set of reviews with the sentiment 1. This means that I would calculate the IDF values based for two documents (because of two classes) where I put together all the reviews of the same class, meaning:

1. Doc1: The text of all reviews with sentiment 0
2. Doc2: The text of all reviews with sentiment 1

I have looked into a few examples and this approach is not what I see done in them.
Approach 2: In the examples for using TF-IDF for text classification, the IDF-values are calculated for each review and not each class (see here), meaning the number of documents is the number of reviews.

Q: Which approach is correct?

My 'pipeline' of classification via tfidf would be:

1. Calculate the IDF with the documents being every review concatenated for each class in all training reviews (like in Doc1 and Doc2)
2. Calculate the TF for every word in each training review
3. Calculate the TF for every word in each testing review
4. Multiply TF*IDF to get the TFIDF-matrix for either train or test

Here is an example in R to illustrate both approaches:
(train_tfidf follows my idea and example_train_tfidf follows approach 2. I calculate the tfidfs, train a model for each approach, make predictions and compare AUCs)

library(text2vec)
library(glmnet)

data("movie_review")
train <- movie_review[1:4000, ]
test <- movie_review[4001:5000, ]

# My idea behind TF-IDF
# create a tfidf-model with one document for each class
create_tfidf_model <- function(train){
  # This is the crucial step for my approach!
  train <- aggregate(review ~ sentiment, data = train, FUN = paste, collapse = " ")
  train_tokens <- itoken(train$review, 
                        preprocessor = tolower, tokenizer = word_tokenizer, ids = train$id, progressbar = FALSE)
  train_vocab <- create_vocabulary(train_tokens, stopwords = stopwords::stopwords("en"))
  vectorizer <- vocab_vectorizer(train_vocab)
  train_dtm <- create_dtm(train_tokens, vectorizer)
  tfidf_model <- TfIdf$new()
  # fit model
  train_tfidf <- fit_transform(train_dtm, tfidf_model)
  return(list(tfidf_model, vectorizer, train_tfidf))
}
# Take a vectorizer and (tf)idf-model and calculate a tfidf-Matrix for data with the idf-weights of the tfidf-model for the words in the vectorizer
create_tfidf_from_model <- function(data, tfidf_model, vectorizer){
  data_tokens <- itoken(data$review, 
                       preprocessor = tolower, tokenizer = word_tokenizer, ids = data$id, progressbar = FALSE)
  data_dtm <- create_dtm(data_tokens, vectorizer)
  data_tfidf <- transform(data_dtm, tfidf_model)
  return(data_tfidf)
}

# Example/Approach 2 from https://cran.r-project.org/web/packages/text2vec/vignettes/text-vectorization.html
tfidf_example <- function(train){
  train_tokens <- itoken(train$review, 
                        preprocessor = tolower, tokenizer = word_tokenizer, ids = train$id, progressbar = FALSE)
  train_vocab <- create_vocabulary(train_tokens, stopwords = stopwords::stopwords("en"))
  vectorizer <- vocab_vectorizer(train_vocab)
  train_dtm <- create_dtm(train_tokens, vectorizer)
  tfidf_model <- TfIdf$new()
  train_tfidf <- fit_transform(train_dtm, tfidf_model)
  return(list(tfidf_model, vectorizer, train_tfidf))
}

## Prediction
tfidf_model <- create_tfidf_model(train)
train_tfidf <- create_tfidf_from_model(train, tfidf_model = tfidf_model[[1]], vectorizer = tfidf_model[[2]])
test_tfidf <- create_tfidf_from_model(test, tfidf_model = tfidf_model[[1]], vectorizer = tfidf_model[[2]])

example_tfidf_model <- tfidf_example(train)
example_train_tfidf <- create_tfidf_from_model(train, tfidf_model = example_tfidf_model[[1]], vectorizer = example_tfidf_model[[2]])
example_test_tfidf <- create_tfidf_from_model(test, tfidf_model = example_tfidf_model[[1]], vectorizer = example_tfidf_model[[2]])


# Train models
glmnet_clas_mine <- cv.glmnet(x = train_tfidf, y = train[['sentiment']], 
                             family = 'binomial', alpha = 1, type.measure = "auc", nfolds = 4, thresh = 1e-3, maxit = 1e3)

glmnet_clas_example <- cv.glmnet(x = example_train_tfidf, y = train[['sentiment']], 
                                family = 'binomial', alpha = 1, type.measure = "auc", nfolds = 4, thresh = 1e-3, maxit = 1e3)

plot(glmnet_clas_mine)
print(paste("max AUC =", round(max(glmnet_clas_mine$cvm), 4)))
plot(glmnet_clas_example)
print(paste("max AUC =", round(max(glmnet_clas_example$cvm), 4)))

# predict with models
pred <- predict(glmnet_clas_mine, test_tfidf, type = 'response')[,1]
example_pred <- predict(glmnet_clas_example, example_test_tfidf, type = 'response')[,1]
# AUC
glmnet:::auc(test$sentiment, pred)
glmnet:::auc(test$sentiment, example_pred)

Evaluating the prediction on the test set gives me an AUC of 0.905 with my approach and an AUC of 0.912 where I count each review as one document. The AUCs are very close to each other. But I my approach is slightly worse.
(The same results holts on the AUC on the train set)

But there is a one thing which I noticed:

# Their sparsity is about equal, but my approach has more values which equal 0
sum(train_tfidf@x != 0)
sum(example_train_tfidf@x != 0)

# This should normally not be done as the matrices are big and sparse
train_tfidf <- as.data.frame(as.matrix(train_tfidf))
example_train_tfidf <- as.data.frame(as.matrix(example_train_tfidf))

# Remove columns where every entry is 0
train_tfidf <-  train_tfidf[, colSums(train_tfidf != 0) > 0]
example_train_tfidf <-  example_train_tfidf[, colSums(example_train_tfidf != 0) > 0]
ncol(train_tfidf)
ncol(example_train_tfidf)

My approach has approximately half the number of features (~20k less) after removing features where every entry is 0 which sounds good to me. But I am still confused why Approach 2 is used in every example. Maybe because it is less code to do and easier to give an example of it?

Answer 1

I believe that the IDF is better to run on each review. Because IDF depends on the index (2 in the case of the class, or N for the number of reviews), you will lose some granularity to the underlying structure of the data. Because even though the class might be the same, the underlying unit is actually the person writing the review, and it makes sense then that this should be your index unit. This is because the reviews are the ones being drawn from the distribution (somewhat iid, but not exactly), and so for a better sense of the idf, we want it measured at the level of the drawing of our sample points.

If you were curious, you could train your model for each of the classes separately, and here you would be assuming that the reviews for the two classes are drawn from separate distributions and are sufficiently different to have different models associated with them. I suspect however that the loss in the extra data will be not worth the gain from specifically modeling each class.