# TF-IDF String to Vector Weka bias

For example, let's say I have a text dataset like:

"words text etc",label
"words text etc",label
"words text etc",label


If I use Weka's String to Word TFIDF transformation, the IDF would consider all the documents in the dataset for IDF.

Wouldn't that make my results biased while classifying the dataset? (since it considers the IDF of the testing portion as well because I made the transform before classifying)

If I split the dataset beforehand, i.e:

Remove Folds 1 --> 10
save as a test file
Remove Folds 1 --> 10 (inverted selection)
save each as a training file


Then if I use StringToWord vector with TF-IDF for the training set and the testing set, I will end up with different vectors in each. How to resolve this?

Am I being too paranoid about bias?

## 1 Answer

You test set should not be included. TF-IDF is a transformation, you should apply it separately to your train and test matrices, after it has been extracted from your train data.

I think this article could help you, here's the relevant excerpt: