For example, let's say I have a text dataset like:
"words text etc",label "words text etc",label "words text etc",label
If I use Weka's String to Word TFIDF transformation, the IDF would consider all the documents in the dataset for IDF.
Wouldn't that make my results biased while classifying the dataset? (since it considers the IDF of the testing portion as well because I made the transform before classifying)
If I split the dataset beforehand, i.e:
Remove Folds 1 --> 10 save as a test file Remove Folds 1 --> 10 (inverted selection) save each as a training file
Then if I use StringToWord vector with TF-IDF for the training set and the testing set, I will end up with different vectors in each. How to resolve this?
Am I being too paranoid about bias?