Merge text collection subsamples for cross-validation

Question

I'm doing a research and to test our methods we're using 5-fold cross-validation. Our collections consist of some pre-classified text. To use the data in Weka, I'm pre-processing it with the StringToWordVector filter. So far, so good...

The problem is: both me and my supervisor are running the algorithms. I use Weka, and he uses his algorithms implemented in Matlab.

In order to use the same subsamples for the 5-fold cross-validation in both Weka and Matlab, I split each of our collections into 5 subsamples. After doing that, for each subsample I created the arff file and applied the StringToWordVector.

The question is: how can I merge these subsamples in order to get the train and test files for each fold?

I can't merge the text files manually and then apply the StringToWordVector, because the words on the training file would be different than the words on the test file. I could also apply the StringToWordVector filter in the collection, without splitting it into subsamples, and then use the StratifiedRemoveFolds filter to generate the train and test files, but I want to use the same subsets on both Weka and Matlab and, if possible, I want to use the subsamples that I've already created.

Answer 1

In case someone else has this problem, Weka has this "classifier" called InputMappedClassifier, which is a wrapper for other classifier, that maps the attributes of the training and test files and then run the selected classifier.