2
$\begingroup$

For example, let's say I have a text dataset like:

"words text etc",label      
"words text etc",label
"words text etc",label

If I use Weka's String to Word TFIDF transformation, the IDF would consider all the documents in the dataset for IDF.

Wouldn't that make my results biased while classifying the dataset? (since it considers the IDF of the testing portion as well because I made the transform before classifying)

If I split the dataset beforehand, i.e:

Remove Folds 1 --> 10 
        save as a test file
Remove Folds 1 --> 10 (inverted selection)
       save each as a training file

Then if I use StringToWord vector with TF-IDF for the training set and the testing set, I will end up with different vectors in each. How to resolve this?

Am I being too paranoid about bias?

$\endgroup$

1 Answer 1

2
$\begingroup$

You test set should not be included. TF-IDF is a transformation, you should apply it separately to your train and test matrices, after it has been extracted from your train data.

I think this article could help you, here's the relevant excerpt:

A subtle point about feature scaling is that it requires knowing feature statistics that we most likely do not know in practice, such as the mean, variance, document frequency, ℓ2 norm, etc. In order to compute the tf-idf representation, we have to compute the inverse document frequencies based on the training data and use these statistics to scale both the training and test data. In scikit-learn, fitting the feature transformer on the training data amounts to collecting the relevant statistics. The fitted transformer can then be applied to the test data.

From Chapter 4. The Effects of Feature Scaling: From Bag-of-Words to Tf-Idf in Feature Engineering for Machine Learning published by O'Rielly press.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.