For example, let's say I have a text dataset like:

"words text etc",label      
"words text etc",label
"words text etc",label

If I use Weka's String to Word TFIDF transformation, the IDF would consider all the documents in the dataset for IDF.

Wouldn't that make my results biased while classifying the dataset? (since it considers the IDF of the testing portion as well because I made the transform before classifying)

If I split the dataset beforehand, i.e:

Remove Folds 1 --> 10 
        save as a test file
Remove Folds 1 --> 10 (inverted selection)
       save each as a training file

Then if I use StringToWord vector with TF-IDF for the training set and the testing set, I will end up with different vectors in each. How to resolve this?

Am I being too paranoid about bias?


You test set should not be included. TF-IDF is a transformation, you should apply it separately to your train and test matrices, after it has been extracted from your train data.

I think this article could help you, here's the relevant excerpt:

enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.